Large Language Models for Mining Biobank-Derived Insights into Health and Disease

Large Language Models for Mining Biobank-Derived Insights into Health and Disease | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Large Language Models for Mining Biobank-Derived Insights into Health and Disease Manuel Corpas, Alfredo Iacoangeli This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6098960/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Large Language Models (LLMs) offer transformative potential for analysing biobank-derived datasets, facilitating knowledge extraction, patient stratification, and predictive modelling. This study benchmarks multiple LLMs in retrieving biomedical insights from a leading biobank, the UK Biobank. UK Biobank-related literature is used as gold standard for assessing coverage and retrieval of some of the best known LLMs, including GPT, Claude, Gemini, Mistral, Llama and DeekSeek. The findings highlight each model’s strengths and limitations, emphasising challenges in data heterogeneity and accessibility. We suggest future research should take advantage of the power of LLMs for enhanced precision in biobank knowledge extraction. Biological sciences/Computational biology and bioinformatics/Data mining Biological sciences/Computational biology and bioinformatics/Databases Biological sciences/Computational biology and bioinformatics/Literature mining Biological sciences/Computational biology and bioinformatics/Machine learning Figures Figure 1 Figure 2 Introduction LLMs such as GPT-based models and their peers are rapidly becoming powerful tools for biomedical research. LLMs are AI systems trained on massive text corpora to understand and generate human-like language 1 . They process and generate text, enabling tasks like summarisation, question answering and hypothesis generation that can be applied to knowledge research 2 . These capabilities are ideally suited to harness the potential offered by biobanks, which are large repositories of biological and health data. The UK Biobank, “the world’s most important health research database” 3 , encompasses extensive environmental, lifestyle, and genetic data on half a million participants 4 , 5 . Numerous discoveries have been powered with UK Biobank data in disease associations and risk factors 6 – 8 . Leveraging the potential that the UK Biobank offers often presents challenges. The data are highly heterogenous, including genomics, clinical records, imaging, questionnaires, etc., with different data types, formats and sizes. Now that DNAnexus has become the gateway to analyse UK Biobank data 9 , the learning curve to understand its contents may be steep, requiring highly specialised bioinformatic knowledge which may pose a barrier for non-technically oriented users. The UK Biobank offers a showcase resource that enables the navigation of existing data types 10 , with summary statistics as well as metadata content. Given the sheer size of the data bank and the complexity of its contents, it is non-trivial to optimise querying, perform analyses and take full advantage of such a unique resource. These hurdles can indeed be surmounted with LLMs. If properly trained, LLMs offer the promise to enhance the production of faster hypotheses, validation of results and the yielding of the tangible benefits for society that a resource such as the UK Biobank promises to serve. This study aims at benchmarking how well current LLMs can be used for the retrieval and summarisation insights from UK Biobank-related literature and metadata. We focus on evaluating the coverage and retrieval capabilities of some of the best known LLMs, including GPT 11 , Claude 12 , Gemini 13 , Mistral 14 , Llama 15 and DeekSeek 16 . Our benchmark analysis sheds light on the dominant themes in UK Biobank-related research and opens new avenues for topics that as of today remain understudied. We believe that large scale biobank data trained and analysed using LLMs will bring significant value and insights into our understanding of health and disease. Methods We tested a set of state-of-the-art general-purpose LLMs. These LLMs were chosen due to their availability and wide usage across different economic sectors. The models that we tested include ChatGPT 4o , ChatGPT o1 , ChatGPT o1 Pro , Claude 3.5 Sonnet , Gemini 2.0 Flash , Mistral Large 2 , Meta Llama 3.1 405B , and DeepSeek’s DeepThink R1 . These models vary in architecture and training data; for instance, some of them are proprietary (e.g., OpenAI’s ChatGPT variants), while others are open source or domain specific (e.g., Meta’s Llama series or DeepSeek’s DeepThink R1). Each of these models were queried under reproducible conditions (i.e., via web interface with the same query each) to ensure a fair comparison in how they retrieve UK Biobank knowledge. No prior explicit training was performed for any of the LLMs tested. Reference Dataset We downloaded from UK Biobank’s showcase 17 all its openly accessible schemas as of 25th January 2025. For our benchmark, we focused on “Schema 19”, which contains the abstracts of peer reviewed publications the resource has compiled to the date. The schema contains the metadata and free text of 8,549 abstracts. This collection spans a wide range of biomedical publications, capturing the diversity of findings emerging from the resource. Key biomedical terms (e.g., specific diseases, methodologies like GWAS) were extracted from these abstracts to serve as points of comparison for the LLM outputs. We also used Schema 27 as source of information from 15,046 applications to the biobank to date. This corpus contains metadata and text related to the titles, applicants and institutions that have been approved for access to the UK Biobank. Evaluation Criteria We defined two main metrics to quantify each model’s performance in retrieving relevant information. “Coverage Score” was used as a measure of breadth of relevant keywords or concepts from the UK Biobank that the LLM’s answer could retrieve. We define this score as the proportion of terms from our reference dataset that appeared in the model’s output for a given query. “Weighted Coverage Score” was another metric we used that refines the coverage by giving more weight to high-impact terms. For example, a concept such as “genome-wide association study (GWAS)” or “mendelian randomization” might be weighted more heavily than other terms that are not as frequent in biobank-related literature. Thus, starting from a set of predefined set of keywords and synonyms (e.g., "Genome-Wide Association Study", "GWAS", "genome-wide association"), we performed embedding-based matching. Coverage Scores were computed by counting matched keywords and Weighted Coverage Scores by summing keyword frequencies (for a detailed explanation of how these scores were calculated see annex). Matching Process For keyword matching we used the sentence transformer model “all-MiniLM-L6-v2” via a python library (“SentenceTransformer”) in order to compute the cosine similarity between keyword embeddings and LLM responses. This allows us to determine whether an LLM’s output contained a reference term (or an equivalent concept). Both the model’s output text and each target term were converted into vector embeddings. We then calculated cosine similarity: if the similarity exceeded a threshold of 0.20, we considered the term “matched” (even if the wording differed). This approach accounts for semantic similarity. For instance, if a model mentioned “heart disease” and the reference term was “cardiovascular disease”, the embedding similarity would likely flag this as a match. A threshold of 0.20 was chosen empirically to balance precision and recall, capturing related terms without being too lenient. False positive matched were manually reviewed to refine this cutoff during development. Prompting Each LLM was prompted with standardised questions derived from common UK Biobank topics. These questions were then typed in all interfaces for the following LLMs: ChatGPT 4o, ChatGPT o1, ChatGPT o1 Pro, Claude 3.5 Sonnet, Gemini 2.0 Flash, Mistral Large 2, Meta Llama 3.1 405B, and DeepSeek’s DeepThink R1. Results were then retrieved and saved as a CSV file for calculation of Coverage Score and Weighted Coverage Score (the CSV files are available in the DATA directory of the GitHub repository for this project 18 ). The questions we used to prompt each LLM were as follows: What is the Subject of the Most Commonly Occurring Keywords in UK Biobank Papers? What is the Subject of the Most Cited Papers Relating to the UK Biobank? What Are the Top 20 Most Prolific Authors Publishing on the UK Biobank? What Are the Top 10 Leading Institutions in Terms of Number of Applications to UK Biobank? Note that these questions rely on a certain degree of interpretation. We would hope not only to get the keywords but also some level of reasoning that would allow us to retrieve dominant topic and insights into UK Biobank’s derived knowledge of health and disease. Results Ground Truth Results First, we present the analysis of the results that we obtained by parsing both Schema 19 (UK Biobank’s 8,549 abstracts) and Schema 27 (UK Biobank’s 15,046 research applications). The code that was used for this is available on GitHub 18 . Figure 1 ( A , B , C , D ) shows the results of calculating the answers from our four prompting questions. Analysis of Top Keywords in Publications The most prevalent keyword was “Humans”, appearing 6,547 times, followed by “Female” (3,469), “Male” (3,277) and “Middle Aged” (2,774) (Fig. 1 A). These results suggest a strong emphasis on human-related studies, with a particular focus on sex and age demographics. Geographical representation was also notable, with “United Kingdom” raking among the top five (2,689 occurrences). Key methodological terms such as “Genome-Wide Association Study” (1,940), “Mendelian Randomization Analysis” (751), and “Prospective Studies” (1,304) indicated a research focus on genetic epidemiology and population-based cohort analyses. Health-related keywords, including “Cardiovascular Diseases” (711) and “Diabetes Mellitus, Type 2” (544), suggest a strong interest in chronic disease genetics. Other notable terms such as “Risk Factors” (2,264), “Genetic Predisposition to Disease” (1,336) and “Multifactorial Inheritance” (508) further highlight the relevance of polygenic risk and complex trait analyses in these studies. Most Cited Articles Related to the UK Biobank An analysis of the most cited research articles associated with the UK Biobank (Fig. 1 B) highlights key areas of scientific interest and impact. The highest-cited publication is on Comparison of Sociodemographic and Health-Related Characteristics of UK Biobank Participants with Those of the General Population (published in the American Journal of Epidemiology ) 19 , which has accumulated 2,548 citations (according the UK Biobank’s metadata schema), underscoring the significance of demographic and health-related factors in large-scale biobank studies. Studies leveraging genome-wide association studies (GWAS) dominate the citation rankings, reflecting the widespread use of the UK Biobank for uncovering genetic risk factors. The second most cited paper, Genome-wide association analyses identifying 44 risk variants ( Nature Genetics ) 20 , has 2,419 citations, followed closely by research on Genome-wide polygenic scores for common diseases ( Nature Genetics , 2,291 citations) 21 . The impact of body composition on mortality is also a prominent topic, with Body-mass index and all-cause mortality ( The Lancet ) accumulating 1,965 citations. Similarly, Gene discovery and polygenic prediction from a GWAS ( Nature Genetics ) 22 has received 1,958 citations, reinforcing the role of biobank-scale datasets in predictive genomics. Mental health research has also gained significant attention, as seen in Genome-wide meta-analysis of depression ( Nature Neuroscience , 1,843 citations) 23 , which highlights the genetic basis of psychiatric conditions. Additionally, Genome-wide meta-analysis identifies new loci and functional pathways influencing Alzheimer’s disease risk ( Nature Genetics , 1,777 citations) 24 and Meta-analysis of GWAS for height ( Human Molecular Genetics , 1,710 citations) 25 further illustrate the diversity of genomic research leveraging the UK Biobank. Beyond genetics, neuroimaging and brain-related studies are also represented, with Multimodal population brain imaging in the UK Biobank prospective study ( Nature Neuroscience ) accumulating 1,577 citations 26 . Finally, Identification of novel risk loci, causal insights, and heritable risk for Parkinson's disease: a meta-analysis of genome-wide association studies ( The Lancet Neurology ) has 1,548 citations 27 . Authorship and Institutional Contributions to UK Biobank Research The analysis of publication output reveals the most prolific researchers contributing to studies leveraging the UK Biobank (Fig. 1 C). George Davey Smith ranks as the most published author, with 122 publications, followed by Naveed Sattar (119), Kari Stefansson (105), and Caroline Hayward (94). Several other notable researchers, including Stephen Burgess (93), Wei Cheng (92), and Carlos Celis-Morales (79), also feature prominently, indicating strong engagement in biobank-based research from diverse fields such as epidemiology, genomics, and cardiometabolic health. In terms of institutional engagement (Fig. 1 D), the University of Oxford leads with 186 applications, reflecting its central role in UK Biobank-related studies. The University of Cambridge (74 applications) and Imperial College London (69 applications) follow, showcasing their strong contributions to biobank-driven investigations. Other major UK institutions, such as University College London (69), University of Edinburgh (62), and University of Manchester (61), demonstrate widespread institutional involvement. Interestingly, UK Biobank Ltd itself appears as a key applicant with 60 applications, likely reflecting in-house research and collaborative projects. Notably, international representation is observed with Sun Yat-Sen University (39 applications), indicating global interest in UK Biobank data. Benchmarking Large Language Models Figure 2 summarises the performance of all tested Large Language Models (LLMs) in retrieving information about UK Biobank research from our reference corpora. We evaluated four specific tasks—covering keywords in publications, top cited papers, top authors, and top applicant institutions—using both Coverage Score (breadth of matched concepts) and Weighted Coverage Score (concepts weighted by their relative importance or frequency). In the case of a score of 0.0 this means the model was not able to retrieve any of the ground truth results. We aggregated each model’s performance across the four tasks to produce an overall benchmark ranking. Keyword Retrieval Figure 2A compares each model’s ability to identify the top 20 most frequent keywords in the UK Biobank literature. Several models (e.g., Meta Llama 3.1 405B and Gemini 2.0 Flash) showed strong performance, with both Coverage and Weighted Coverage Scores above 0.70. Mistral Large 2 also achieved high coverage (0.70) but displayed a slightly lower weighted score (0.76), indicating good overall breadth in matching keywords, though some high-frequency terms were missed or underrepresented. Models such as DeepThink R1 and Claude 3.5 Sonnet performed moderately (Coverage around 0.45–0.50) but still captured a substantial fraction of key biomedical concepts. Top Cited Papers Figure 2B displays results for identifying the most highly cited articles based on our curated list. Only a few LLMs produced meaningful coverage here. Gemini 2.0 Flash performed best, capturing 20% of the article top 10 references (Coverage Score = 0.20) with a Weighted Coverage Score of 0.25. ChatGPT 0.1 pro and Meta Llama 3.1 405B also recognised a subset of these seminal articles, albeit with lower coverage. Other models had no success (Coverage and Weighted Coverage 0.00), suggesting that explicit references to these top-cited publications were not retrieved effectively in their outputs. Most Prolific Authors Figure 2C evaluates retrieval of the top 20 UK Biobank authors by publication count. Mistral Large 2 led on both coverage (0.70) and weighted coverage (0.69), correctly listing many of the highest-output authors such as George Davey Smith, Naveed Sattar, and Kari Stefansson. Claude 3.5 Sonnet followed, reaching around 0.60 coverage, while Gemini 2.0 Flash captured a more moderate 0.45 coverage. Notably, several ChatGPT variants (0.1 and 0.1 pro) had negligible coverage (0.00), indicating that these models did not generate or match author names reliably under the tested prompts. Top Applicant Institutions Figure 2D focuses on the top 10 institutions with the highest number of UK Biobank applications. Gemini 2.0 Flash stood out again as the strongest performer, with Coverage Score = 0.90 and Weighted Coverage = 0.92, meaning it identified nearly all these institutions along with their higher-frequency mentions. Three ChatGPT variants ( 4o, 0.1, 0.1 pro ) also demonstrated robust coverage (0.80–0.82), while DeepThink R1 had a moderate 0.70 coverage. Claude 3.5 Sonnet and Mistral Large 2 attained lower coverage (0.40–0.30), and Meta Llama 3.1 405B trailed at 0.10 coverage. Overall Accuracy Ranking We computed each model’s average Weighted Coverage across all four tasks to generate an overall ranking (Fig. 2E). Gemini 2.0 Flash led with an average Weighted Coverage of 0.611, reflecting consistently high performance on keywords, top cited papers, authors, and institutions. DeepThink R1 placed second (0.436), followed by Mistral Large 2 (0.424) and ChatGPT 0.1 pro (0.419). Claude 3.5 Sonnet, ChatGPT 0.1, and ChatGPT 4o formed a middle tier (0.370–0.367), whereas Meta Llama 3.1 405B had the lowest average Weighted Coverage of 0.328 among the tested models. Discussion Analyses of UK Biobank publications highlight two dominant themes: demography-focused research and genetics-driven explorations. The frequent appearance of keywords such as “Female,” “Male,” “Middle Aged,” and “United Kingdom” underscores the ongoing efforts to characterise population-level patterns across sex, age, and geography. Simultaneously, high-frequency methodological terms ( “Genome-Wide Association Study” , “Mendelian Randomization Analysis” ) demonstrate a focus on uncovering genetic and causal determinants of complex diseases. Citation patterns for the top papers reinforce these observations, showing that demographic profiling and GWAS-based insights remain highly influential within the research community. The presence of “ Body-mass index ” and “ all-cause mortality ” as a leading topic signals a strong interest in modifiable risk factors, while prominent attention to mental health (e.g., depression) and neuroimaging suggests a diversification beyond classic epidemiological themes. Examining prolific authors and leading institutions reveals a robust UK-based leadership (e.g., University of Oxford, University of Cambridge), but also demonstrates global engagement with UK Biobank data, as evidenced by applications from institutions like Sun Yat-Sen University. This growing international interest likely reflects the versatility of the UK Biobank for addressing a broad range of research questions, from cardiometabolic conditions to psychiatric phenotypes. Gemini 2.0 Flash Emerged as the Top Performer Across All Tasks In this study, we have evaluated a diverse set of Large Language Models (LLMs) on their ability to retrieve and summarise knowledge from UK Biobank–related literature. Our benchmarking tasks focused on four distinct domains—keyword identification, top-cited papers, top authors, and leading institutions—and were assessed using Coverage and Weighted Coverage Scores. Gemini 2.0 Flash emerged as the top performer across all tasks, demonstrating consistently high retrieval of both common and high-impact terms (e.g., frequently cited authors, institutions, and conceptual keywords). Meanwhile, DeepThink R1, Mistral Large 2, and ChatGPT 0.1 Pro formed the next tier, producing relatively comprehensive outputs but missing some high-frequency or highly weighted elements. Models such as Meta Llama 3.1 405B performed least well under our evaluation, suggesting limited coverage for UK Biobank–specific queries. Strong Efforts Researching the Polygenic Architecture of Common Diseases Our analysis underscores several key insights: LLMs can capture important fields reflecting the core themes of UK Biobank research, including genome-wide association studies (GWAS), risk factors, phenotypes, cardiovascular disease, and Type 2 diabetes. Recurrent keywords such as “Humans,” “Female,” and “Male” highlight the demographic orientation of large-scale cohort studies, whereas terms like “multifactorial inheritance” and “genetic predisposition” confirm growing interest in the polygenic architecture of common diseases. The most prolific authors in the corpus (e.g., George Davey Smith, Naveed Sattar, Kari Stefansson) emerged with high consistency in better-performing models, illustrating how LLMs can efficiently surface individual research profiles within large academic corpora. Institutional analyses demonstrated that organisations closely tied to the UK Biobank (e.g., University of Oxford, UK Biobank Ltd.) were prominently identified by the strongest models, confirming that LLMs can detect usage patterns indicating institutional research engagement. Limitations Despite these strengths, there are several limitations to consider. First, our evaluation hinges on coverage metrics which, while useful for quantifying breadth, do not fully capture the accuracy, coherence, or contextual relevance of the responses. Indeed, while an AI model might correctly mention a relevant keyword or an author's name in its response, it might not actually use it in a meaningful way. For example, if a model is asked about key topics in UK Biobank research, it might list “genome-wide association studies” (GWAS) as a keyword but fail to explain what it means, why it is important, or how it relates to UK Biobank findings. Secondly, our approach relies on predefined keywords and synonyms, potentially missing relevant concepts expressed in alternative scientific terminology. Furthermore, LLMs are stochastic by nature and can produce different responses across runs. This variability may lower reproducibility for certain queries. Additionally, the study does not compare LLM performance against a baseline model that randomly selects keywords, authors, or institutions from general biomedical literature. Without this comparison, we cannot determine whether the LLMs are truly capturing UK Biobank-specific patterns or if their outputs reflect general biomedical trends. A naive guesser, for example, could randomly select frequently occurring biomedical terms without any true understanding of the UK Biobank dataset. If an LLM does not significantly outperform such a naive model, it suggests that the LLM may not be extracting meaningful UK Biobank-specific insights. Therefore, without this baseline comparison, the specificity of the LLMs' performance to UK Biobank remains uncertain. Finally, we did not assess tasks such as clinical summarisation accuracy, hypothesis generation, or multimodal reasoning, which are critical frontiers for advanced biobank-based analytics but beyond our current scope. Future Work In light of these findings, future work should refine LLM evaluation frameworks to incorporate precision (i.e., whether spurious keywords appear) and recall (i.e., whether key concepts are missed), as well as deeper assessments of the LLMs’ capacity for constructing cohesive narratives or drawing new inferences from biobank data. Aligning LLM outputs more closely with clinically or biologically significant outcomes will be important for making these models genuinely impactful in population-scale health research. Ultimately, as LLMs evolve and domain-specific fine-tuning becomes more accessible, these systems have the potential to streamline literature mining, expand hypothesis discovery, and enhance collaboration across the broad user base of the UK Biobank. Conclusions This benchmarking study demonstrates that Large Language Models (LLMs) can effectively retrieve and summarise key elements of UK Biobank-related literature, spanning keywords, top-cited papers, prolific authors, and research-active institutions. While certain models - especially Gemini 2.0 Flash - stand out for their higher coverage and better handling of high-impact topics, others show variability in capturing domain-specific concepts or relevant contextual details. The multifactorial nature of UK Biobank data demands advanced NLP solutions to navigate genomics, epidemiology, phenotypic measurements, and diverse clinical aspects. This evaluation highlights the value of using Weighted Coverage metrics to distinguish between general coverage and prioritisation of impactful terms. However, improved methodologies incorporating precision, recall, and expert validation will further refine how LLMs can assist biobank research. Ultimately, integrating LLMs into biobank analytics can help overcome the cost and time challenges typically associated with analysing such vast datasets. By automating literature analysis and surfacing hidden relationships, these models have the potential to accelerate big-picture insights, hypothesis generation, and translational applications. Continued advancements in both LLM architectures and domain-specific training approaches will further enhance the comprehensiveness and reliability of automated biobank literature mining, benefiting researchers, clinicians, and society at large. Declarations Data and Code Availability All “ground truth” data were retrieved from the UK Biobank Showcase ( https://biobank.ndph.ox.ac.uk/showcase/ ). All scripts and data-processing routines used in this study are publicly accessible via GitHub: https://github.com/manuelcorpas/LLM_4_UKB 18 . This repository contains the code for benchmarking Large Language Models, keyword identification, citation analysis, and the evaluation pipeline described in this paper. The results from LLM queries that were subsequently analysed for calculating our coverage metrics are available as a CSV files in the DATA directory of the repository. Author Contribution MC designed and performed the work and wrote the initial draft. AI contributed with improvements and discussions to the submitted paper. All authors reviewed the manuscript. Acknowledgements We acknowledge the UK Biobank for providing the extensive datasets and metadata that made this benchmarking study possible. Their commitment to open and collaborative research continues to advance our understanding of health and disease on a global scale. References Kumar, P. Large language models (LLMs): survey, technical frameworks, and future challenges. Artif Intell Rev 57, 260 (2024). Zhang, K. et al. Revolutionizing Health Care: The Transformative Impact of Large Language Models in Medicine. J Med Internet Res 27, e59069 (2025). UK Biobank - UK Biobank. https://www.ukbiobank.ac.uk (2025). About our data. UK Biobank https://www.ukbiobank.ac.uk/enable-your-research/about-our-data (2024). Bycroft, C. et al. The UK Biobank resource with deep phenotyping and genomic data. Nature 562, 203–209 (2018). Garg, M. et al. Disease prediction with multi-omics and biomarkers empowers case–control genetic discoveries in the UK Biobank. Nat Genet 56, 1821–1831 (2024). Sinnott-Armstrong, N. et al. Genetics of 35 blood and urine biomarkers in the UK Biobank. Nat Genet 53, 185–194 (2021). Gadd, D. A. et al. Blood protein assessment of leading incident diseases and mortality in the UK Biobank. Nat Aging 4, 939–948 (2024). UK Biobank Research Analysis Platform. https://www.ukbiobank.ac.uk/enable-your-research/research-analysis-platform (2024). : Showcase Homepage. https://biobank.ndph.ox.ac.uk/showcase/ . ChatGPT. https://chatgpt.com. Claude. https://claude.ai/new. ‎Gemini – chat to supercharge your ideas. Gemini https://gemini.google.com . Le Chat. Mistral AI https://chat.mistral.ai . Llama 3.1 405B Online Chat | ChatHub. https://chathub.gg/models/meta/llama3.1-405b . DeepSeek. https://chat.deepseek.com. : Schema. https://biobank.ndph.ox.ac.uk/showcase/schema.cgi . manuelcorpas. manuelcorpas/LLM_4_UKB. (2025). Fry, A. et al. Comparison of Sociodemographic and Health-Related Characteristics of UK Biobank Participants With Those of the General Population. American Journal of Epidemiology 186, 1026–1034 (2017). Wray, N. R. et al. Genome-wide association analyses identify 44 risk variants and refine the genetic architecture of major depression. Nat Genet 50, 668–681 (2018). Khera, A. V. et al. Genome-wide polygenic scores for common diseases identify individuals with risk equivalent to monogenic mutations. Nat Genet 50, 1219–1224 (2018). Lee, J. J. et al. Gene discovery and polygenic prediction from a genome-wide association study of educational attainment in 1.1 million individuals. Nat Genet 50, 1112–1121 (2018). Howard, D. M. et al. Genome-wide meta-analysis of depression identifies 102 independent variants and highlights the importance of the prefrontal brain regions. Nat Neurosci 22, 343–352 (2019). Jansen, I. E. et al. Genome-wide meta-analysis identifies new loci and functional pathways influencing Alzheimer’s disease risk. Nat Genet 51, 404–413 (2019). Yengo, L. et al. Meta-analysis of genome-wide association studies for height and body mass index in ∼700000 individuals of European ancestry. Human Molecular Genetics 27, 3641–3649 (2018). Miller, K. L. et al. Multimodal population brain imaging in the UK Biobank prospective epidemiological study. Nat Neurosci 19, 1523–1536 (2016). Nalls, M. A. et al. Identification of novel risk loci, causal insights, and heritable risk for Parkinson’s disease: a meta-analysis of genome-wide association studies. The Lancet Neurology 18, 1091–1102 (2019). Additional Declarations No competing interests reported. Supplementary Files Annex.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6098960","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":421505286,"identity":"162725fd-7ecd-498e-adfe-9bd3b5a5341a","order_by":0,"name":"Manuel Corpas","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABBUlEQVRIiWNgGAWjYDADfgglgSTEQ0CLZAOQOADSwkasFoMDYC0MhLXoNjA/fFy5w87e+Eby4c8fcyzy5Oc3H/vAUGPHYHDmAFYtZgfYjA3PnklO3HYjLU3i4DaJYoNjbMkzGI4lMxicbcChhYdNsrGNOcHsdo4ZA1BL4gY2HmOg6w4wGJzH7jColnp749n5nz+AtMxv4//MwPCPoJbDjBukcxhADktsOMbDzMDYdgC3ww4D/dJ45njijPvPzCTOghx2LM2YIbEvmUcSl/ePNz982Lij2p6/5/DjD5Xb6hLnNx9+zPDhm50c35kE7C5jBmJGDBckEIpITC2jYBSMglEwCpAAAFuRXr32uwdeAAAAAElFTkSuQmCC","orcid":"","institution":"University of Westminster","correspondingAuthor":true,"prefix":"","firstName":"Manuel","middleName":"","lastName":"Corpas","suffix":""},{"id":421505287,"identity":"1e937833-3eca-45c3-941f-0598d7231c58","order_by":1,"name":"Alfredo Iacoangeli","email":"","orcid":"","institution":"King's College London","correspondingAuthor":false,"prefix":"","firstName":"Alfredo","middleName":"","lastName":"Iacoangeli","suffix":""}],"badges":[],"createdAt":"2025-02-24 17:23:11","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6098960/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6098960/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":78146101,"identity":"ba77e4f2-56cf-440d-9bce-4deb63cd91c7","added_by":"auto","created_at":"2025-03-10 10:55:07","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":360882,"visible":true,"origin":"","legend":"\u003cp\u003eBenchmarking Results from UK Biobank Schema Data. (A) \u003cem\u003eTop Keywords in Publications:\u003c/em\u003e Identifies the most frequently cited keywords in UK Biobank papers, revealing key research themes such as GWAS, cardiovascular disease, type 2 diabetes, and Mendelian randomisation. (B) \u003cem\u003eMost Cited Articles:\u003c/em\u003e Highlights the top publications associated with the UK Biobank, including prominent studies on demographic profiling, GWAS findings, and polygenic risk scores. (C) \u003cem\u003eProlific Authors:\u003c/em\u003e Summarises the authors with the highest publication counts (e.g., George Davey Smith, Naveed Sattar, Kari Stefansson), reflecting the major contributors to biobank-related research. (D) \u003cem\u003eLeading Applicant Institutions:\u003c/em\u003e Shows the top institutions by number of research applications, with the University of Oxford (the coordinating institution of the UK Biobank) ranking first.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-6098960/v1/b1db134f3c0d628f8f7c1606.png"},{"id":78146095,"identity":"1b7bfd73-c703-491f-ab65-189d52f56ebb","added_by":"auto","created_at":"2025-03-10 10:55:06","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":433799,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eComposite Results of LLM Benchmarking on UK Biobank-Related Queries.\u003c/strong\u003e \u003cstrong\u003e(A)\u003c/strong\u003e \u003cem\u003eKeyword Coverage:\u003c/em\u003eThe bar chart shows how each model (x-axis) performs in identifying the top 20 most frequently used UK Biobank keywords. The height of each bar indicates the Coverage Score (blue) and Weighted Coverage Score (orange) for each model. \u003cstrong\u003e(B)\u003c/strong\u003e \u003cem\u003eTop Cited Papers:\u003c/em\u003e Models are evaluated on their ability to retrieve the top-cited UK Biobank publications. Higher bars mean a higher fraction of influential articles recognised, with heavier weighting given to those with more citations. \u003cstrong\u003e(C)\u003c/strong\u003e \u003cem\u003eMost Prolific Authors:\u003c/em\u003e Each model’s output is assessed for matching the top 20 authors by publication count. Coverage Scores capture how many authors each LLM mentions, while Weighted Coverage Scores emphasise authors with more total publications. \u003cstrong\u003e(D)\u003c/strong\u003e \u003cem\u003eApplicant Institutions:\u003c/em\u003e Shows performance in detecting the 10 most frequent UK Biobank applicant institutions. Scores are weighted according to the institution’s application counts. \u003cstrong\u003e(E)\u003c/strong\u003e \u003cem\u003eOverall Model Ranking:\u003c/em\u003e An aggregated average Weighted Coverage Score across all tasks (A–D). This consolidates model performance into a single ranking, identifying those most effective at retrieving UK Biobank-related knowledge.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-6098960/v1/9bf40a55ddf2dff6e5b8ad40.png"},{"id":78149117,"identity":"0ed04618-9de7-42fa-9e92-0f7671538c37","added_by":"auto","created_at":"2025-03-10 11:27:07","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1508998,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6098960/v1/ee85d339-4857-4e38-bb02-dc374897029e.pdf"},{"id":78146093,"identity":"235ed347-fba3-4b11-b872-c847135f8280","added_by":"auto","created_at":"2025-03-10 10:55:06","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":19183,"visible":true,"origin":"","legend":"","description":"","filename":"Annex.docx","url":"https://assets-eu.researchsquare.com/files/rs-6098960/v1/86fada65547401e021cd2613.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Large Language Models for Mining Biobank-Derived Insights into Health and Disease","fulltext":[{"header":"Introduction","content":"\u003cp\u003eLLMs such as GPT-based models and their peers are rapidly becoming powerful tools for biomedical research. LLMs are AI systems trained on massive text corpora to understand and generate human-like language \u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u003c/sup\u003e. They process and generate text, enabling tasks like summarisation, question answering and hypothesis generation that can be applied to knowledge research \u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e. These capabilities are ideally suited to harness the potential offered by biobanks, which are large repositories of biological and health data. The UK Biobank, “the world’s most important health research database” \u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e, encompasses extensive environmental, lifestyle, and genetic data on half a million participants \u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e,\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e. Numerous discoveries have been powered with UK Biobank data in disease associations and risk factors \u003csup\u003e\u003cspan additionalcitationids=\"CR7\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e–\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eLeveraging the potential that the UK Biobank offers often presents challenges. The data are highly heterogenous, including genomics, clinical records, imaging, questionnaires, etc., with different data types, formats and sizes. Now that DNAnexus has become the gateway to analyse UK Biobank data \u003csup\u003e\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e, the learning curve to understand its contents may be steep, requiring highly specialised bioinformatic knowledge which may pose a barrier for non-technically oriented users. The UK Biobank offers a showcase resource that enables the navigation of existing data types \u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e, with summary statistics as well as metadata content. Given the sheer size of the data bank and the complexity of its contents, it is non-trivial to optimise querying, perform analyses and take full advantage of such a unique resource. These hurdles can indeed be surmounted with LLMs. If properly trained, LLMs offer the promise to enhance the production of faster hypotheses, validation of results and the yielding of the tangible benefits for society that a resource such as the UK Biobank promises to serve.\u003c/p\u003e \u003cp\u003eThis study aims at benchmarking how well current LLMs can be used for the retrieval and summarisation insights from UK Biobank-related literature and metadata. We focus on evaluating the coverage and retrieval capabilities of some of the best known LLMs, including GPT \u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e, Claude \u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e, Gemini \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e, Mistral \u003csup\u003e\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e, Llama \u003csup\u003e\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e and DeekSeek \u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e. Our benchmark analysis sheds light on the dominant themes in UK Biobank-related research and opens new avenues for topics that as of today remain understudied. We believe that large scale biobank data trained and analysed using LLMs will bring significant value and insights into our understanding of health and disease.\u003c/p\u003e \n\n \n\n "},{"header":"Methods","content":"\u003cp\u003eWe tested a set of state-of-the-art general-purpose LLMs. These LLMs were chosen due to their availability and wide usage across different economic sectors. The models that we tested include \u003cem\u003eChatGPT 4o\u003c/em\u003e, \u003cem\u003eChatGPT o1\u003c/em\u003e, \u003cem\u003eChatGPT o1 Pro\u003c/em\u003e, \u003cem\u003eClaude 3.5 Sonnet\u003c/em\u003e, \u003cem\u003eGemini 2.0 Flash\u003c/em\u003e, \u003cem\u003eMistral Large 2\u003c/em\u003e, \u003cem\u003eMeta Llama 3.1 405B\u003c/em\u003e, and \u003cem\u003eDeepSeek’s DeepThink R1\u003c/em\u003e. These models vary in architecture and training data; for instance, some of them are proprietary (e.g., OpenAI’s ChatGPT variants), while others are open source or domain specific (e.g., Meta’s Llama series or DeepSeek’s DeepThink R1). Each of these models were queried under reproducible conditions (i.e., via web interface with the same query each) to ensure a fair comparison in how they retrieve UK Biobank knowledge. No prior explicit training was performed for any of the LLMs tested.\u003c/p\u003e\u003cp\u003e \u003cb\u003eReference Dataset\u003c/b\u003e \u003c/p\u003e\u003cp\u003eWe downloaded from UK Biobank’s showcase \u003csup\u003e\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e all its openly accessible schemas as of 25th January 2025. For our benchmark, we focused on “Schema 19”, which contains the abstracts of peer reviewed publications the resource has compiled to the date. The schema contains the metadata and free text of 8,549 abstracts. This collection spans a wide range of biomedical publications, capturing the diversity of findings emerging from the resource. Key biomedical terms (e.g., specific diseases, methodologies like GWAS) were extracted from these abstracts to serve as points of comparison for the LLM outputs. We also used Schema 27 as source of information from 15,046 applications to the biobank to date. This corpus contains metadata and text related to the titles, applicants and institutions that have been approved for access to the UK Biobank.\u003c/p\u003e\u003ch3\u003eEvaluation Criteria\u003c/h3\u003e\u003cp\u003eWe defined two main metrics to quantify each model’s performance in retrieving relevant information. “Coverage Score” was used as a measure of breadth of relevant keywords or concepts from the UK Biobank that the LLM’s answer could retrieve. We define this score as the proportion of terms from our reference dataset that appeared in the model’s output for a given query. “Weighted Coverage Score” was another metric we used that refines the coverage by giving more weight to high-impact terms. For example, a concept such as “genome-wide association study (GWAS)” or “mendelian randomization” might be weighted more heavily than other terms that are not as frequent in biobank-related literature. Thus, starting from a set of predefined set of keywords and synonyms (e.g., \"Genome-Wide Association Study\", \"GWAS\", \"genome-wide association\"), we performed embedding-based matching. Coverage Scores were computed by counting matched keywords and Weighted Coverage Scores by summing keyword frequencies (for a detailed explanation of how these scores were calculated see annex).\u003c/p\u003e\u003ch2\u003eMatching Process\u003c/h2\u003e\u003cp\u003eFor keyword matching we used the sentence transformer model \u003cem\u003e“all-MiniLM-L6-v2”\u003c/em\u003e via a python library (“SentenceTransformer”) in order to compute the cosine similarity between keyword embeddings and LLM responses. This allows us to determine whether an LLM’s output contained a reference term (or an equivalent concept). Both the model’s output text and each target term were converted into vector embeddings. We then calculated cosine similarity: if the similarity exceeded a threshold of 0.20, we considered the term “matched” (even if the wording differed). This approach accounts for semantic similarity. For instance, if a model mentioned “heart disease” and the reference term was “cardiovascular disease”, the embedding similarity would likely flag this as a match. A threshold of 0.20 was chosen empirically to balance precision and recall, capturing related terms without being too lenient. False positive matched were manually reviewed to refine this cutoff during development.\u003c/p\u003e\u003ch3\u003ePrompting\u003c/h3\u003e\u003cp\u003eEach LLM was prompted with standardised questions derived from common UK Biobank topics. These questions were then typed in all interfaces for the following LLMs: ChatGPT 4o, ChatGPT o1, ChatGPT o1 Pro, Claude 3.5 Sonnet, Gemini 2.0 Flash, Mistral Large 2, Meta Llama 3.1 405B, and DeepSeek’s DeepThink R1. Results were then retrieved and saved as a CSV file for calculation of Coverage Score and Weighted Coverage Score (the CSV files are available in the DATA directory of the GitHub repository for this project \u003csup\u003e\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e). The questions we used to prompt each LLM were as follows:\u003c/p\u003e\u003cp\u003e \u003c/p\u003e\u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhat is the Subject of the Most Commonly Occurring \u003cb\u003eKeywords\u003c/b\u003e in UK Biobank Papers?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhat is the Subject of the Most Cited \u003cb\u003ePapers\u003c/b\u003e Relating to the UK Biobank?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhat Are the Top 20 Most Prolific \u003cb\u003eAuthors\u003c/b\u003e Publishing on the UK Biobank?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhat Are the Top 10 Leading \u003cb\u003eInstitutions\u003c/b\u003e in Terms of Number of Applications to UK Biobank?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eNote that these questions rely on a certain degree of interpretation. We would hope not only to get the keywords but also some level of reasoning that would allow us to retrieve dominant topic and insights into UK Biobank’s derived knowledge of health and disease.\u003c/p\u003e"},{"header":"Results","content":"\u003cdiv id=\"Sec6\" class=\"Section2\"\u003e\n\u003ch2\u003eGround Truth Results\u003c/h2\u003e\n\u003cp\u003eFirst, we present the analysis of the results that we obtained by parsing both Schema 19 (UK Biobank\u0026rsquo;s 8,549 abstracts) and Schema 27 (UK Biobank\u0026rsquo;s 15,046 research applications). The code that was used for this is available on GitHub \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e. Figure\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e (\u003cstrong\u003eA\u003c/strong\u003e, \u003cstrong\u003eB\u003c/strong\u003e, \u003cstrong\u003eC\u003c/strong\u003e, \u003cstrong\u003eD\u003c/strong\u003e) shows the results of calculating the answers from our four prompting questions.\u003c/p\u003e\n\u003c/div\u003e\n\u003ch3\u003eAnalysis of Top Keywords in Publications\u003c/h3\u003e\n\u003cp\u003eThe most prevalent keyword was \u0026ldquo;Humans\u0026rdquo;, appearing 6,547 times, followed by \u0026ldquo;Female\u0026rdquo; (3,469), \u0026ldquo;Male\u0026rdquo; (3,277) and \u0026ldquo;Middle Aged\u0026rdquo; (2,774) (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003eA). These results suggest a strong emphasis on human-related studies, with a particular focus on sex and age demographics.\u003c/p\u003e\n\u003cp\u003eGeographical representation was also notable, with \u0026ldquo;United Kingdom\u0026rdquo; raking among the top five (2,689 occurrences). Key methodological terms such as \u0026ldquo;Genome-Wide Association Study\u0026rdquo; (1,940), \u0026ldquo;Mendelian Randomization Analysis\u0026rdquo; (751), and \u0026ldquo;Prospective Studies\u0026rdquo; (1,304) indicated a research focus on genetic epidemiology and population-based cohort analyses.\u003c/p\u003e\n\u003cp\u003eHealth-related keywords, including \u0026ldquo;Cardiovascular Diseases\u0026rdquo; (711) and \u0026ldquo;Diabetes Mellitus, Type 2\u0026rdquo; (544), suggest a strong interest in chronic disease genetics. Other notable terms such as \u0026ldquo;Risk Factors\u0026rdquo; (2,264), \u0026ldquo;Genetic Predisposition to Disease\u0026rdquo; (1,336) and \u0026ldquo;Multifactorial Inheritance\u0026rdquo; (508) further highlight the relevance of polygenic risk and complex trait analyses in these studies.\u003c/p\u003e\n\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e\n\u003ch2\u003eMost Cited Articles Related to the UK Biobank\u003c/h2\u003e\n\u003cp\u003eAn analysis of the most cited research articles associated with the UK Biobank (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003eB) highlights key areas of scientific interest and impact. The highest-cited publication is on \u003cem\u003eComparison of Sociodemographic and Health-Related Characteristics of UK Biobank Participants with Those of the General Population\u003c/em\u003e (published in the \u003cem\u003eAmerican Journal of Epidemiology\u003c/em\u003e) \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e, which has accumulated 2,548 citations (according the UK Biobank\u0026rsquo;s metadata schema), underscoring the significance of demographic and health-related factors in large-scale biobank studies.\u003c/p\u003e\n\u003cp\u003eStudies leveraging genome-wide association studies (GWAS) dominate the citation rankings, reflecting the widespread use of the UK Biobank for uncovering genetic risk factors. The second most cited paper, \u003cem\u003eGenome-wide association analyses identifying 44 risk variants\u003c/em\u003e (\u003cem\u003eNature Genetics\u003c/em\u003e) \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e, has 2,419 citations, followed closely by research on \u003cem\u003eGenome-wide polygenic scores for common diseases\u003c/em\u003e (\u003cem\u003eNature Genetics\u003c/em\u003e, 2,291 citations) \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eThe impact of body composition on mortality is also a prominent topic, with \u003cem\u003eBody-mass index and all-cause mortality\u003c/em\u003e (\u003cem\u003eThe Lancet\u003c/em\u003e) accumulating 1,965 citations. Similarly, \u003cem\u003eGene discovery and polygenic prediction from a GWAS\u003c/em\u003e (\u003cem\u003eNature Genetics\u003c/em\u003e) \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e has received 1,958 citations, reinforcing the role of biobank-scale datasets in predictive genomics.\u003c/p\u003e\n\u003cp\u003eMental health research has also gained significant attention, as seen in \u003cem\u003eGenome-wide meta-analysis of depression\u003c/em\u003e (\u003cem\u003eNature Neuroscience\u003c/em\u003e, 1,843 citations) \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e, which highlights the genetic basis of psychiatric conditions. Additionally, \u003cem\u003eGenome-wide meta-analysis identifies new loci and functional pathways influencing Alzheimer\u0026rsquo;s disease risk\u003c/em\u003e (\u003cem\u003eNature Genetics\u003c/em\u003e, 1,777 citations) \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e and \u003cem\u003eMeta-analysis of GWAS for height\u003c/em\u003e (\u003cem\u003eHuman Molecular Genetics\u003c/em\u003e, 1,710 citations) \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e further illustrate the diversity of genomic research leveraging the UK Biobank.\u003c/p\u003e\n\u003cp\u003eBeyond genetics, neuroimaging and brain-related studies are also represented, with \u003cem\u003eMultimodal population brain imaging in the UK Biobank prospective study\u003c/em\u003e (\u003cem\u003eNature Neuroscience\u003c/em\u003e) accumulating 1,577 citations \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e26\u003c/span\u003e\u003c/sup\u003e. Finally, \u003cem\u003eIdentification of novel risk loci, causal insights, and heritable risk for Parkinson's disease: a meta-analysis of genome-wide association studies\u003c/em\u003e (\u003cem\u003eThe Lancet Neurology\u003c/em\u003e) has 1,548 citations \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\n\u003c/div\u003e\n\u003ch3\u003eAuthorship and Institutional Contributions to UK Biobank Research\u003c/h3\u003e\n\u003cp\u003eThe analysis of publication output reveals the most prolific researchers contributing to studies leveraging the UK Biobank (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003eC). George Davey Smith ranks as the most published author, with 122 publications, followed by Naveed Sattar (119), Kari Stefansson (105), and Caroline Hayward (94). Several other notable researchers, including Stephen Burgess (93), Wei Cheng (92), and Carlos Celis-Morales (79), also feature prominently, indicating strong engagement in biobank-based research from diverse fields such as epidemiology, genomics, and cardiometabolic health.\u003c/p\u003e\n\u003cp\u003eIn terms of institutional engagement (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003eD), the University of Oxford leads with 186 applications, reflecting its central role in UK Biobank-related studies. The University of Cambridge (74 applications) and Imperial College London (69 applications) follow, showcasing their strong contributions to biobank-driven investigations. Other major UK institutions, such as University College London (69), University of Edinburgh (62), and University of Manchester (61), demonstrate widespread institutional involvement.\u003c/p\u003e\n\u003cp\u003eInterestingly, UK Biobank Ltd itself appears as a key applicant with 60 applications, likely reflecting in-house research and collaborative projects. Notably, international representation is observed with Sun Yat-Sen University (39 applications), indicating global interest in UK Biobank data.\u003c/p\u003e\n\u003ch3\u003eBenchmarking Large Language Models\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eFigure 2\u003c/strong\u003e summarises the performance of all tested Large Language Models (LLMs) in retrieving information about UK Biobank research from our reference corpora. We evaluated four specific tasks\u0026mdash;covering keywords in publications, top cited papers, top authors, and top applicant institutions\u0026mdash;using both Coverage Score (breadth of matched concepts) and Weighted Coverage Score (concepts weighted by their relative importance or frequency). In the case of a score of 0.0 this means the model was not able to retrieve any of the ground truth results. We aggregated each model\u0026rsquo;s performance across the four tasks to produce an overall benchmark ranking.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eKeyword Retrieval\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFigure 2A\u003c/strong\u003e compares each model\u0026rsquo;s ability to identify the top 20 most frequent keywords in the UK Biobank literature. Several models (e.g., Meta Llama 3.1 405B and Gemini 2.0 Flash) showed strong performance, with both Coverage and Weighted Coverage Scores above 0.70. Mistral Large 2 also achieved high coverage (0.70) but displayed a slightly lower weighted score (0.76), indicating good overall breadth in matching keywords, though some high-frequency terms were missed or underrepresented. Models such as DeepThink R1 and Claude 3.5 Sonnet performed moderately (Coverage around 0.45\u0026ndash;0.50) but still captured a substantial fraction of key biomedical concepts.\u003c/p\u003e\n\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e\n\u003ch2\u003eTop Cited Papers\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eFigure 2B\u003c/strong\u003e displays results for identifying the most highly cited articles based on our curated list. Only a few LLMs produced meaningful coverage here. Gemini 2.0 Flash performed best, capturing 20% of the article top 10 references (Coverage Score\u0026thinsp;=\u0026thinsp;0.20) with a Weighted Coverage Score of 0.25. ChatGPT 0.1 pro and Meta Llama 3.1 405B also recognised a subset of these seminal articles, albeit with lower coverage. Other models had no success (Coverage and Weighted Coverage 0.00), suggesting that explicit references to these top-cited publications were not retrieved effectively in their outputs.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec12\" class=\"Section2\"\u003e\n\u003ch2\u003eMost Prolific Authors\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eFigure 2C\u003c/strong\u003e evaluates retrieval of the top 20 UK Biobank authors by publication count. Mistral Large 2 led on both coverage (0.70) and weighted coverage (0.69), correctly listing many of the highest-output authors such as George Davey Smith, Naveed Sattar, and Kari Stefansson. Claude 3.5 Sonnet followed, reaching around 0.60 coverage, while Gemini 2.0 Flash captured a more moderate 0.45 coverage. Notably, several ChatGPT variants (0.1 and 0.1 pro) had negligible coverage (0.00), indicating that these models did not generate or match author names reliably under the tested prompts.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec13\" class=\"Section2\"\u003e\n\u003ch2\u003eTop Applicant Institutions\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eFigure 2D\u003c/strong\u003e focuses on the top 10 institutions with the highest number of UK Biobank applications. Gemini 2.0 Flash stood out again as the strongest performer, with Coverage Score\u0026thinsp;=\u0026thinsp;0.90 and Weighted Coverage\u0026thinsp;=\u0026thinsp;0.92, meaning it identified nearly all these institutions along with their higher-frequency mentions. Three ChatGPT variants (\u003cem\u003e4o, 0.1, 0.1 pro\u003c/em\u003e) also demonstrated robust coverage (0.80\u0026ndash;0.82), while DeepThink R1 had a moderate 0.70 coverage. Claude 3.5 Sonnet and Mistral Large 2 attained lower coverage (0.40\u0026ndash;0.30), and Meta Llama 3.1 405B trailed at 0.10 coverage.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec14\" class=\"Section2\"\u003e\n\u003ch2\u003eOverall Accuracy Ranking\u003c/h2\u003e\n\u003cp\u003eWe computed each model\u0026rsquo;s average Weighted Coverage across all four tasks to generate an overall ranking (Fig.\u0026nbsp;2E). Gemini 2.0 Flash led with an average Weighted Coverage of 0.611, reflecting consistently high performance on keywords, top cited papers, authors, and institutions. DeepThink R1 placed second (0.436), followed by Mistral Large 2 (0.424) and ChatGPT 0.1 pro (0.419). Claude 3.5 Sonnet, ChatGPT 0.1, and ChatGPT 4o formed a middle tier (0.370\u0026ndash;0.367), whereas Meta Llama 3.1 405B had the lowest average Weighted Coverage of 0.328 among the tested models.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eAnalyses of UK Biobank publications highlight two dominant themes: demography-focused research and genetics-driven explorations. The frequent appearance of keywords such as \u003cem\u003e\u0026ldquo;Female,\u0026rdquo; \u0026ldquo;Male,\u0026rdquo; \u0026ldquo;Middle Aged,\u0026rdquo;\u003c/em\u003e and \u003cem\u003e\u0026ldquo;United Kingdom\u0026rdquo;\u003c/em\u003e underscores the ongoing efforts to characterise population-level patterns across sex, age, and geography. Simultaneously, high-frequency methodological terms (\u003cem\u003e\u0026ldquo;Genome-Wide Association Study\u0026rdquo;\u003c/em\u003e, \u003cem\u003e\u0026ldquo;Mendelian Randomization Analysis\u0026rdquo;\u003c/em\u003e) demonstrate a focus on uncovering genetic and causal determinants of complex diseases.\u003c/p\u003e \u003cp\u003eCitation patterns for the top papers reinforce these observations, showing that demographic profiling and GWAS-based insights remain highly influential within the research community. The presence of \u0026ldquo;\u003cem\u003eBody-mass index\u003c/em\u003e\u0026rdquo; and \u0026ldquo;\u003cem\u003eall-cause mortality\u003c/em\u003e\u0026rdquo; as a leading topic signals a strong interest in modifiable risk factors, while prominent attention to mental health (e.g., depression) and neuroimaging suggests a diversification beyond classic epidemiological themes.\u003c/p\u003e \u003cp\u003eExamining prolific authors and leading institutions reveals a robust UK-based leadership (e.g., University of Oxford, University of Cambridge), but also demonstrates global engagement with UK Biobank data, as evidenced by applications from institutions like Sun Yat-Sen University. This growing international interest likely reflects the versatility of the UK Biobank for addressing a broad range of research questions, from cardiometabolic conditions to psychiatric phenotypes.\u003c/p\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003eGemini 2.0 Flash Emerged as the Top Performer Across All Tasks\u003c/h2\u003e \u003cp\u003eIn this study, we have evaluated a diverse set of Large Language Models (LLMs) on their ability to retrieve and summarise knowledge from UK Biobank\u0026ndash;related literature. Our benchmarking tasks focused on four distinct domains\u0026mdash;keyword identification, top-cited papers, top authors, and leading institutions\u0026mdash;and were assessed using Coverage and Weighted Coverage Scores. Gemini 2.0 Flash emerged as the top performer across all tasks, demonstrating consistently high retrieval of both common and high-impact terms (e.g., frequently cited authors, institutions, and conceptual keywords). Meanwhile, DeepThink R1, Mistral Large 2, and ChatGPT 0.1 Pro formed the next tier, producing relatively comprehensive outputs but missing some high-frequency or highly weighted elements. Models such as Meta Llama 3.1 405B performed least well under our evaluation, suggesting limited coverage for UK Biobank\u0026ndash;specific queries.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003eStrong Efforts Researching the Polygenic Architecture of Common Diseases\u003c/h2\u003e \u003cp\u003eOur analysis underscores several key insights:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eLLMs can capture important fields reflecting the core themes of UK Biobank research, including genome-wide association studies (GWAS), risk factors, phenotypes, cardiovascular disease, and Type 2 diabetes.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eRecurrent keywords such as \u0026ldquo;Humans,\u0026rdquo; \u0026ldquo;Female,\u0026rdquo; and \u0026ldquo;Male\u0026rdquo; highlight the demographic orientation of large-scale cohort studies, whereas terms like \u0026ldquo;multifactorial inheritance\u0026rdquo; and \u0026ldquo;genetic predisposition\u0026rdquo; confirm growing interest in the polygenic architecture of common diseases.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eThe most prolific authors in the corpus (e.g., George Davey Smith, Naveed Sattar, Kari Stefansson) emerged with high consistency in better-performing models, illustrating how LLMs can efficiently surface individual research profiles within large academic corpora.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eInstitutional analyses demonstrated that organisations closely tied to the UK Biobank (e.g., University of Oxford, UK Biobank Ltd.) were prominently identified by the strongest models, confirming that LLMs can detect usage patterns indicating institutional research engagement.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003eLimitations\u003c/h2\u003e \u003cp\u003eDespite these strengths, there are several limitations to consider. First, our evaluation hinges on coverage metrics which, while useful for quantifying breadth, do not fully capture the accuracy, coherence, or contextual relevance of the responses. Indeed, while an AI model might correctly mention a relevant keyword or an author's name in its response, it might not actually use it in a meaningful way. For example, if a model is asked about key topics in UK Biobank research, it might list \u0026ldquo;genome-wide association studies\u0026rdquo; (GWAS) as a keyword but fail to explain what it means, why it is important, or how it relates to UK Biobank findings. Secondly, our approach relies on predefined keywords and synonyms, potentially missing relevant concepts expressed in alternative scientific terminology. Furthermore, LLMs are stochastic by nature and can produce different responses across runs. This variability may lower reproducibility for certain queries. Additionally, the study does not compare LLM performance against a baseline model that randomly selects keywords, authors, or institutions from general biomedical literature. Without this comparison, we cannot determine whether the LLMs are truly capturing UK Biobank-specific patterns or if their outputs reflect general biomedical trends. A naive guesser, for example, could randomly select frequently occurring biomedical terms without any true understanding of the UK Biobank dataset. If an LLM does not significantly outperform such a naive model, it suggests that the LLM may not be extracting meaningful UK Biobank-specific insights. Therefore, without this baseline comparison, the specificity of the LLMs' performance to UK Biobank remains uncertain. Finally, we did not assess tasks such as clinical summarisation accuracy, hypothesis generation, or multimodal reasoning, which are critical frontiers for advanced biobank-based analytics but beyond our current scope.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec19\" class=\"Section2\"\u003e \u003ch2\u003eFuture Work\u003c/h2\u003e \u003cp\u003eIn light of these findings, future work should refine LLM evaluation frameworks to incorporate precision (i.e., whether spurious keywords appear) and recall (i.e., whether key concepts are missed), as well as deeper assessments of the LLMs\u0026rsquo; capacity for constructing cohesive narratives or drawing new inferences from biobank data. Aligning LLM outputs more closely with clinically or biologically significant outcomes will be important for making these models genuinely impactful in population-scale health research. Ultimately, as LLMs evolve and domain-specific fine-tuning becomes more accessible, these systems have the potential to streamline literature mining, expand hypothesis discovery, and enhance collaboration across the broad user base of the UK Biobank.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusions","content":"\u003cp\u003eThis benchmarking study demonstrates that Large Language Models (LLMs) can effectively retrieve and summarise key elements of UK Biobank-related literature, spanning keywords, top-cited papers, prolific authors, and research-active institutions. While certain models - especially Gemini 2.0 Flash - stand out for their higher coverage and better handling of high-impact topics, others show variability in capturing domain-specific concepts or relevant contextual details.\u003c/p\u003e \u003cp\u003eThe multifactorial nature of UK Biobank data demands advanced NLP solutions to navigate genomics, epidemiology, phenotypic measurements, and diverse clinical aspects. This evaluation highlights the value of using Weighted Coverage metrics to distinguish between general coverage and prioritisation of impactful terms. However, improved methodologies incorporating precision, recall, and expert validation will further refine how LLMs can assist biobank research.\u003c/p\u003e \u003cp\u003eUltimately, integrating LLMs into biobank analytics can help overcome the cost and time challenges typically associated with analysing such vast datasets. By automating literature analysis and surfacing hidden relationships, these models have the potential to accelerate big-picture insights, hypothesis generation, and translational applications. Continued advancements in both LLM architectures and domain-specific training approaches will further enhance the comprehensiveness and reliability of automated biobank literature mining, benefiting researchers, clinicians, and society at large.\u003c/p\u003e "},{"header":"Declarations","content":"\u003cdiv id=\"Sec21\" class=\"Section2\"\u003e \u003ch2\u003eData and Code Availability\u003c/h2\u003e \u003cp\u003eAll \u0026ldquo;ground truth\u0026rdquo; data were retrieved from the UK Biobank Showcase (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://biobank.ndph.ox.ac.uk/showcase/\u003c/span\u003e\u003cspan address=\"https://biobank.ndph.ox.ac.uk/showcase/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e). All scripts and data-processing routines used in this study are publicly accessible via GitHub: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/manuelcorpas/LLM_4_UKB\u003c/span\u003e\u003cspan address=\"https://github.com/manuelcorpas/LLM_4_UKB\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e \u003csup\u003e\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e. This repository contains the code for benchmarking Large Language Models, keyword identification, citation analysis, and the evaluation pipeline described in this paper. The results from LLM queries that were subsequently analysed for calculating our coverage metrics are available as a CSV files in the DATA directory of the repository.\u003c/p\u003e \u003c/div\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eMC designed and performed the work and wrote the initial draft. AI contributed with improvements and discussions to the submitted paper. All authors reviewed the manuscript.\u003c/p\u003e\u003ch2\u003eAcknowledgements\u003c/h2\u003e \u003cp\u003eWe acknowledge the UK Biobank for providing the extensive datasets and metadata that made this benchmarking study possible. Their commitment to open and collaborative research continues to advance our understanding of health and disease on a global scale.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eKumar, P. Large language models (LLMs): survey, technical frameworks, and future challenges. Artif Intell Rev 57, 260 (2024).\u003c/li\u003e\n\u003cli\u003eZhang, K. \u003cem\u003eet al.\u003c/em\u003e Revolutionizing Health Care: The Transformative Impact of Large Language Models in Medicine. J Med Internet Res 27, e59069 (2025).\u003c/li\u003e\n\u003cli\u003eUK Biobank - UK Biobank. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.ukbiobank.ac.uk\u003c/span\u003e\u003c/span\u003e (2025).\u003c/li\u003e\n\u003cli\u003eAbout our data. UK Biobank \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.ukbiobank.ac.uk/enable-your-research/about-our-data\u003c/span\u003e\u003c/span\u003e (2024).\u003c/li\u003e\n\u003cli\u003eBycroft, C. \u003cem\u003eet al.\u003c/em\u003e The UK Biobank resource with deep phenotyping and genomic data. Nature 562, 203\u0026ndash;209 (2018).\u003c/li\u003e\n\u003cli\u003eGarg, M. \u003cem\u003eet al.\u003c/em\u003e Disease prediction with multi-omics and biomarkers empowers case\u0026ndash;control genetic discoveries in the UK Biobank. Nat Genet 56, 1821\u0026ndash;1831 (2024).\u003c/li\u003e\n\u003cli\u003eSinnott-Armstrong, N. \u003cem\u003eet al.\u003c/em\u003e Genetics of 35 blood and urine biomarkers in the UK Biobank. Nat Genet 53, 185\u0026ndash;194 (2021).\u003c/li\u003e\n\u003cli\u003eGadd, D. A. \u003cem\u003eet al.\u003c/em\u003e Blood protein assessment of leading incident diseases and mortality in the UK Biobank. Nat Aging 4, 939\u0026ndash;948 (2024).\u003c/li\u003e\n\u003cli\u003eUK Biobank Research Analysis Platform. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.ukbiobank.ac.uk/enable-your-research/research-analysis-platform\u003c/span\u003e\u003c/span\u003e (2024).\u003c/li\u003e\n\u003cli\u003e: Showcase Homepage. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://biobank.ndph.ox.ac.uk/showcase/\u003c/span\u003e\u003c/span\u003e.\u003c/li\u003e\n\u003cli\u003eChatGPT. https://chatgpt.com.\u003c/li\u003e\n\u003cli\u003eClaude. https://claude.ai/new.\u003c/li\u003e\n\u003cli\u003e\u0026lrm;Gemini \u0026ndash; chat to supercharge your ideas. \u003cem\u003eGemini\u003c/em\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://gemini.google.com\u003c/span\u003e\u003c/span\u003e.\u003c/li\u003e\n\u003cli\u003eLe Chat. \u003cem\u003eMistral AI\u003c/em\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://chat.mistral.ai\u003c/span\u003e\u003c/span\u003e.\u003c/li\u003e\n\u003cli\u003eLlama 3.1 405B Online Chat | ChatHub. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://chathub.gg/models/meta/llama3.1-405b\u003c/span\u003e\u003c/span\u003e.\u003c/li\u003e\n\u003cli\u003eDeepSeek. https://chat.deepseek.com.\u003c/li\u003e\n\u003cli\u003e: Schema. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://biobank.ndph.ox.ac.uk/showcase/schema.cgi\u003c/span\u003e\u003c/span\u003e.\u003c/li\u003e\n\u003cli\u003emanuelcorpas. manuelcorpas/LLM_4_UKB. (2025).\u003c/li\u003e\n\u003cli\u003eFry, A. \u003cem\u003eet al.\u003c/em\u003e Comparison of Sociodemographic and Health-Related Characteristics of UK Biobank Participants With Those of the General Population. American Journal of Epidemiology 186, 1026\u0026ndash;1034 (2017).\u003c/li\u003e\n\u003cli\u003eWray, N. R. \u003cem\u003eet al.\u003c/em\u003e Genome-wide association analyses identify 44 risk variants and refine the genetic architecture of major depression. Nat Genet 50, 668\u0026ndash;681 (2018).\u003c/li\u003e\n\u003cli\u003eKhera, A. V. \u003cem\u003eet al.\u003c/em\u003e Genome-wide polygenic scores for common diseases identify individuals with risk equivalent to monogenic mutations. Nat Genet 50, 1219\u0026ndash;1224 (2018).\u003c/li\u003e\n\u003cli\u003eLee, J. J. \u003cem\u003eet al.\u003c/em\u003e Gene discovery and polygenic prediction from a genome-wide association study of educational attainment in 1.1 million individuals. Nat Genet 50, 1112\u0026ndash;1121 (2018).\u003c/li\u003e\n\u003cli\u003eHoward, D. M. \u003cem\u003eet al.\u003c/em\u003e Genome-wide meta-analysis of depression identifies 102 independent variants and highlights the importance of the prefrontal brain regions. Nat Neurosci 22, 343\u0026ndash;352 (2019).\u003c/li\u003e\n\u003cli\u003eJansen, I. E. \u003cem\u003eet al.\u003c/em\u003e Genome-wide meta-analysis identifies new loci and functional pathways influencing Alzheimer\u0026rsquo;s disease risk. Nat Genet 51, 404\u0026ndash;413 (2019).\u003c/li\u003e\n\u003cli\u003eYengo, L. \u003cem\u003eet al.\u003c/em\u003e Meta-analysis of genome-wide association studies for height and body mass index in \u0026sim;700000 individuals of European ancestry. Human Molecular Genetics 27, 3641\u0026ndash;3649 (2018).\u003c/li\u003e\n\u003cli\u003eMiller, K. L. \u003cem\u003eet al.\u003c/em\u003e Multimodal population brain imaging in the UK Biobank prospective epidemiological study. Nat Neurosci 19, 1523\u0026ndash;1536 (2016).\u003c/li\u003e\n\u003cli\u003eNalls, M. A. \u003cem\u003eet al.\u003c/em\u003e Identification of novel risk loci, causal insights, and heritable risk for Parkinson\u0026rsquo;s disease: a meta-analysis of genome-wide association studies. \u003cem\u003eThe Lancet Neurology\u003c/em\u003e 18, 1091\u0026ndash;1102 (2019).\u0026nbsp;\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-6098960/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6098960/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eLarge Language Models (LLMs) offer transformative potential for analysing biobank-derived datasets, facilitating knowledge extraction, patient stratification, and predictive modelling. This study benchmarks multiple LLMs in retrieving biomedical insights from a leading biobank, the UK Biobank. UK Biobank-related literature is used as gold standard for assessing coverage and retrieval of some of the best known LLMs, including GPT, Claude, Gemini, Mistral, Llama and DeekSeek. The findings highlight each model\u0026rsquo;s strengths and limitations, emphasising challenges in data heterogeneity and accessibility. We suggest future research should take advantage of the power of LLMs for enhanced precision in biobank knowledge extraction.\u003c/p\u003e","manuscriptTitle":"Large Language Models for Mining Biobank-Derived Insights into Health and Disease","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-03-10 10:55:02","doi":"10.21203/rs.3.rs-6098960/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"f6f3abc4-a599-4ca7-9dc0-a1e0034194ba","owner":[],"postedDate":"March 10th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":44930082,"name":"Biological sciences/Computational biology and bioinformatics/Data mining"},{"id":44930083,"name":"Biological sciences/Computational biology and bioinformatics/Databases"},{"id":44930084,"name":"Biological sciences/Computational biology and bioinformatics/Literature mining"},{"id":44930085,"name":"Biological sciences/Computational biology and bioinformatics/Machine learning"}],"tags":[],"updatedAt":"2025-03-10T10:55:04+00:00","versionOfRecord":[],"versionCreatedAt":"2025-03-10 10:55:02","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6098960","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6098960","identity":"rs-6098960","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}