Careful design of Large Language Model pipelines enables expert-level retrieval of evidence-based information from conservation syntheses

Careful design of Large Language Model pipelines enables expert-level retrieval of evidence-based information from conservation syntheses | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Careful design of Large Language Model pipelines enables expert-level retrieval of evidence-based information from conservation syntheses Radhika Iyer, Alec Christie, Anil Madhavapeddy, Sam Reynolds, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5409185/v2 This work is licensed under a CC BY 4.0 License Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Abstract Wise use of evidence to support efficient conservation action is key to tackling biodiversity loss with limited time and resources. Evidence syntheses provide key recommendations for conservation decision-makers by assessing and summarising evidence, but are not always easy to access, digest, and use. Recent advances in Large Language Models (LLMs) present both opportunities and risks in enabling faster and more intuitive systems to access evidence syntheses and databases. Such systems for natural language search and open-ended evidence-based responses are pipelines comprising many components. Most critical of these components are the LLM used and how evidence is retrieved from the database. We evaluate the performance of ten LLMs across six different database retrieval strategies against human experts in answering synthetic multiple-choice question exams on the effects of conservation interventions using the Conservation Evidence database. We found that LLM performance was comparable with human experts over 45 filtered questions, both in correctly answering them and retrieving the document used to generate them. Across 1867 unfiltered questions, LLM performance demonstrated a level of conservation-specific knowledge, but this varied across topic areas. A hybrid retrieval strategy that combines keywords and vector embeddings performed best by a substantial margin. We also tested against a state-of-the-art previous generation LLM which was outperformed by all ten current models - including smaller, cheaper models. Our findings suggest that, with careful domain-specific design, LLMs could potentially be powerful tools for enabling expert-level use of evidence syntheses and databases. However, general LLMs used ‘out-of-the-box’ are likely to perform poorly and misinform decision-makers. By establishing that LLMs exhibit comparable performance with human synthesis experts on providing restricted responses to queries of evidence syntheses and databases, future work can build on our approach to quantify LLM performance in providing open-ended responses. Biological sciences/Ecology/Conservation biology Earth and environmental sciences/Ecology/Conservation biology Scientific community and society/Scientific community/Policy Social science/Science, technology and society Scientific community and society/Social sciences/Decision making large language models retrieval augmented generation retrieval evidence evidence-based conservation decision support Figures Figure 1 Figure 2 Introduction To maximise our chances of bending the curve of biodiversity loss 1 with limited time and resources, we need to use the best available evidence to increase the effectiveness of conservation efforts whilst also reducing harmful impacts to both nature and local communities 2 – 5 . Whilst evidence syntheses are key to informing more effective practice and policy by distilling the scientific literature 6 – 8 , they are not always easy for decision-makers to access, digest and use 9 – 11 . However, recent advances in the capabilities of Large Language Models (LLMs; Box 1) have made it easier than ever to gain access to summaries of information and on-demand answers to specific questions across a variety of domains 12 – 15 . LLMs clearly have considerable potential to summarise information efficiently from vast corpuses of literature, but their use in rigorously supporting effective decision-making remains controversial and fraught with potential pitfalls and dangers, including the risk of misinformation from biases and errors 16 , 17 . There is particular concern that the proliferation of natural language interfaces or ‘chatbots’ (e.g., OpenAI’s ChatGPT , Microsoft Co-Pilot, and Google’s Gemini) could misinform decision-makers who may be attracted to LLMs for their ability to rapidly and flexibly answer their questions 18 – 23 , particularly in crisis disciplines such as biodiversity conservation 24 . LLM usage by those providing advice to decision-makers (e.g., consultants and scientists) could also further compound poor quality decision-making without properly considering the quality and relevance of evidence 16 , 21 , 25 , 26 . For example, LLMs used ‘out-of-the-box’ have the potential for hallucinations, false references and citations, out-of-date and biased information based on inherent biases in the training data used to build these models 27 . It is therefore vital to determine whether reliable implementations of LLMs that derive their information from robust evidence syntheses and databases can be developed that enable rapid access to evidence bases without misinforming decision-makers. Box 1 – Glossary and explanation of study data source and methodology Large Language Models (LLMs) are neural networks trained on large quantities of text. Given a block of text, called the context , they are optimised to predict the small piece of text that follows. Adding this small piece of text to the context and repeating the process enables text generation. With tuning these models can follow instructions and carry out multi-turn conversations, exemplified in ChatGPT . Context is the initial set of text given to a LLM that contains instructions and potentially data for it to use. In the Transformer model used by all state-of-the-art models (and all tested in this paper) the computational resources required scales quadratically with context window. This makes responses to long contexts expensive and is the motivation behind techniques for only including relevant information in the context. Retrieval-Augmented Generation (RAG) is one such technique for identifying relevant information to populate the context. In this approach the user query is used to retrieve potentially relevant documents from a corpus, and these are then added to the context along with the user query and sent to the LLM . The Conservation Evidence database ( www.conservationevidence.com ) contains over 8800 studies, at time of writing, testing the effectiveness of over 3800 conservation interventions on biodiversity outcomes (of which 2,250 have studies testing them). It is structured into topic areas called synopses (e.g., Amphibian, Peatland, and Bird Conservation), within which study summaries are organised under the different actions they test. For each action, key message paragraphs summarise the findings of studies and background paragraphs provide key contextual information describing the conservation action being considered. Each action is then assessed using structured expert elicitation to provide an effectiveness category that accounts for the potential effectiveness and harms of the action and the certainty of the evidence. A LLM (Claude 3.5 Sonnet) was used to generate a multiple-choice exam for each of the 2,250 actions using an automated method 30 , (Panel A and B in the figure below) excluding questions that solely asked questions based on the effectiveness categories in the Conservation Evidence databases. This formed a larger set of 1867 unfiltered questions. We also refined these down to a filtered set of 45 questions to enable a comparison with human experts, ensuring that filtered questions were clear and could be answered with a single, accurate answer. Ten LLMs were then asked to provide answers under six different exam conditions for each question within the unfiltered and filtered sets (Panel B in figure below). These exam conditions included three different types of retrieval strategies: sparse, dense, and hybrid retrieval. Sparse retrieval uses keyword-based metrics to find relevant documents from amongst a corpus. This can work well when there is significant keyword overlap between queries and documents but can fail when semantically related documents do not share keywords. Dense retrieval uses a small language model to embed documents or smaller chunks of them as a point in a high dimensional space. These models are optimised to minimise the distance of semantically related text in the embedding space. The user query is embedded and the closest documents in the corpus are returned as relevant. Hybrid retrieval uses a small language model called a reranker to compare documents retrieved from other methods with the query. This can increase performance as the reranking model has access to the full text of potentially relevant documents. Few studies have quantitatively evaluated the reliability of using LLMs for decision support 12 , 18 and none, to our knowledge, in conservation have made comparisons with human experts. It is also unclear how recent advances in fine-tuning and text retrieval techniques (e.g., Retrieval Augmented Generation: RAG; Box 1) to improve the performance of LLMs in specific contexts and reduce errors may help to minimise the risks of misinforming decision-makers 28 , 29 . We also lack an understanding of the subject-specific information that different LLMs possess across different topic areas and therefore whether certain LLMs might provide more reliable decision support for certain fields. Before evaluating how LLMs perform at providing open-ended, free-text responses to questions, we first need to understand their performance at providing restricted responses to questions and retrieving the correct information. We expect LLM performance to decline as tasks become more complex and so we first need to establish where performance issues occur. Once a performance threshold is determined, we can design systems accordingly to ensure the risk of misinforming decision-makers is minimised, with comparable or lower levels of error than humans. Here we begin this process by first testing LLM performance on relatively simple tasks with restricted responses. We ask ten different state-of-the-art LLMs to provide answers to conservation-related multiple-choice questions, including identifying the relevant source document used to answer each question. We use the Conservation Evidence database as a case study , asking LLMs questions on the effects of conservation interventions for which evidence exists in this database (Box 1). We adapt and apply an automated method 30 to measure the task-specific accuracy of Retrieval-Augmented LLMs using automatically generated synthetic exams composed of multiple-choice questions based on a given corpus of documents (Box 1; Methods). This represents a cost-efficient, interpretable, and robust strategy to select optimal components for an initial RAG system providing an intelligent search function for users of the Conservation Evidence database . To determine its robustness to errors, we ask whether each LLM performs better at answering questions and retrieving the correct source document to answer questions than: 1. random guessing; 2. human experts in conservation evidence synthesis; 3. other LLMs; and 4. a predecessor of one of the LLMs - i.e., has there been improvement over time? Our approach could be applied across fields and disciplines to evaluate the suitability of different LLMs and retrieval strategies for providing decision support on specific subjects. Methods We evaluated the performance of different Large Language Models (LLMs) in answering questions based on the evidence contained within the Conservation Evidence database as a case study . Our methodology was adapted from a previously published automated approach 30 , which is designed to measure the task-specific accuracy of Retrieval-Augmented Generation (RAG) LLMs using automatically generated synthetic exams composed of multiple-choice questions based on a given corpus of documents (Box 1). Conservation Evidence Corpus The Conservation Evidence database comprises a collation of studies testing the effects of conservation interventions on biodiversity outcomes for 26 topic areas (called synopses) structured around various species and habitat groups (e.g., Birds, Bees, Terrestrial Mammals). Each synopsis contains a range of actions (22 to 400 per synopsis), with each action including an action title and number, background information, key messages, and evidence summaries – this text constituted a ‘document’ for the purposes of our study (Fig. S1), if any evidence has been found to test that action. For this study, we focused on the 2,250 actions that had at least one associated study. See Figures S1 and S2 for examples of summaries of the evidence for different actions. Exam Generation We used Claude-3.5 Sonnet for exam question generation, as it was the strongest publicly available LLM at the time of the study. The exam generation process involved several steps: Corpus Preparation : We reduced the action documents to key messages and removed actions without supporting evidence. Question Generation : We generated 3–4 questions per action using Claude-3.5 Sonnet (see Table S1 for prompt). Filtering Process : We applied several filters to ensure question quality: Shuffled answer options to prevent bias. Removed questions referring directly to actions (for closed-book testing). Applied Jaccard similarity thresholds to ensure we generated high quality incorrect answers (or discriminators 30 ) – i.e., we removed questions that contained multiple rephrased correct answers (intra-candidate similarity), or where the phrasing of the correct answer reflected that of the question, giving away the answer (extra-candidate similarity). Removed questions solely based on effectiveness ratings from the CE database (e.g., "How effective is .."). We deviated from an automated evaluation method 30 by not applying Item Response Theory for iterative question filtering due to using all post-filtering questions across evaluated LLMs. This gave a large dataset of 1867 unfiltered questions, which whilst potentially containing inaccuracies, provided a basis for comparing relative performance across different LLMs, retrieval strategies, and conservation topic areas (Box 1). Human Evaluation We also curated a filtered dataset of 45 questions (checked for clarity and accuracy – see below) to measure the absolute performance of different LLMs and retrieval strategies and provide a human evaluation benchmark. To produce this filtered dataset, we: Randomly sampled three actions from each synopsis. Applied automated filtering as described above. Iterated the process to ensure each synopsis had at least one question. Manually filtered the questions down to a final subset of 45 questions by two reviewers, who were researchers from the Conservation Evidence project team and did not take part in answering the survey questions. The filtering was based on three main criteria: 1. the clarity of the question, 2. the accuracy of the answers, and 3. that there was one clear correct answer. Both reviewers independently selected questions to reject and then met to discuss any disagreements. The final subset of 45 questions were then reviewed again by both researchers to check they met all three criteria. To establish a human evaluation benchmark, we formed a human expert comparison group that represented the entire Conservation Evidence project team (six experts) based at the University of Cambridge who were uniquely qualified to answer questions and retrieve information on the Conservation Evidence database . Their experience in using the online database ( www.conservationevidence.com ) to retrieve information to answer practitioner questions on conservation interventions provided the ideal human comparison for this evaluation – i.e., the benchmark they would provide would represent highly expert use and querying of the Conservation Evidence database to compare LLMs and retrieval strategies against, which could not be achieved by sourcing experts from a more general group of conservation experts. Each human expert answered all 45 multiple choice questions using an online anonymous survey implemented via Qualtrics survey software. The questions recorded the responses of these participants to these questions and asked them to provide a link to the action webpage (source document) that they used to answer the question (i.e., where they retrieved the evidence from). We also timed how long it took each participant to answer each question using a built-in timing function in Qualtrics survey software. We use median time and interquartile range to be conservative as there were some extreme values, probably caused by participants taking breaks during questions. We obtained ethical approval from the University of Cambridge Computing Science Ethics Committee (review no. #2324). Please see the Participant Information Sheet and Consent Forms in the Supporting Information. Retrieval Strategies We evaluated three retrieval strategies (Box 1) that are commonly used in RAG systems to retrieve relevant information: Dense Retrieval : We used Nomic Embed Text v1.5 through the SentenceTransformers (SBERT) module for embedding. This method represents documents and queries as dense vectors in a high-dimensional space, allowing for semantic similarity comparisons. Sparse Retrieval : We employed BM25, a probabilistic retrieval function that ranks documents based on the appearance of query terms, considering term frequency and document length. Hybrid Retrieval : We used a cross-encoder (ms-marco-MiniLM-L-6-v2) for re-ranking. This approach combines the strengths of both dense and sparse retrieval, using an initial retrieval step followed by a more computationally intensive re-ranking step. All retrieval strategies selected two action source documents that were added to the prompt sent to the LLM (Table S1). LLM Evaluation We evaluated the following state-of-the-art LLMs at the time of conducting the study: Llama 3.1 8B Instruct-Turbo (FP8) Llama 3.1 70B Instruct-Turbo (FP8) Gemma2 Instruct − 9B (BF16) Gemma2 Instruct − 27B (BF16) Mixtral 8x22B Instruct (BF16) Gemini 1.5 Flash (gemini-1.5-flash-001) Gemini 1.5 Pro (gemini-1.5-pro-001) Claude 3.5 Sonnet (claude-3-5-sonnet-20240620) GPT -4o ( gpt -4o-2024-08-06) GPT -4o mini ( gpt -4o-mini-2024-07-18). We used Google Cloud’s Vertex AI for Gemini 1.5 Pro and Gemini 1.5 Flash, OpenAI for GPT -4o and GPT4o mini, Claude for Claude 3.5 Sonnet and Deepinfra for all other models. To maximise the repeatability of LLM responses, we set the temperature hyperparameter to the minimum value of 0 – this effectively minimises the randomness of LLM responses and maximises predictability as much as possible (i.e., almost deterministic). We tested these LLMs under six different exam conditions reflecting different retrieval strategies (or lack of them): Closed book (no source documents provided) Oracle (using the singular source document used to generate the question) Confused (oracle document plus one random document from an action with evidence in the same synopsis) Dense retrieval Sparse retrieval Hybrid retrieval (top 5 dense + top 5 sparse + reranking to top 2). See Table S1 for the prompt structure used and examples of prompts. Statistical testing First, we tested the null hypotheses that there was no difference in the accuracy of each LLM (using a hybrid retrieval strategy) and a random guesser (with a 25% chance) at correctly answering the 45 multiple-choice questions. We used a modified permutation test, with a similar approach to a sign test, whereby we directly compared the correct and incorrect answers given by each LLM versus a random guesser. For questions where the random guesser gave a correct answer and the LLM gave an incorrect answer, a value of -1 was assigned. For the converse, a value of + 1 was assigned (i.e. positive values = LLM wins, negative values = random guesser wins). For cases where both the LLM and random guesser got the questions right or wrong (i.e., a draw), a value of 0 was assigned. As there was a 25% chance of there being a draw for each question, we subtracted the number of draws that might have occurred by chance across the 45 questions – this was done by randomly sampling from the numbers 0 and 1 (0 = no draw, 1 = draw, with a probability of 0.75 of no draw and 0.25 of a draw) with replacement 45 times and taking the sum. We calculated a test statistic by taking the sum of the − 1s, +1s, and remaining 0s across the 45 questions as in a conventional sign test. This process was repeated 10,000 times to produce a raw p-value by calculating the proportion of test statistics equal to zero. Since we tested ten null hypotheses (one for each LLM ) separately, we used the Holm adjustment to correct p-values for multiplicity. We also calculated the mean, standard deviation, and 95% Confidence Intervals of the permutation test statistic. Second, we repeated the permutation test described previously to test the null hypotheses that there was no difference in the accuracy of each LLM and a randomly selected human expert at answering multiple-choice questions. We used paired data for both six human experts and each LLM (using a hybrid retrieval strategy) for all the 45 filtered questions. For each question, we randomly selected one of the six human expert’s answers, comparing its correctness to the correctness of the given LLM ’s answer. This process was repeated 1,000,000 times given the large number (6 45 ) of possible combinations. Third, we repeated the permutation test for the null hypotheses that there was no difference in the retrieval accuracy of each of the three retrieval strategies (sparse, dense, and hybrid) and a randomly selected human expert. As the number of possible retrieval answers was large (2250 potential action pages to retrieve), we did not need to adjust for the probability of draws by chance as we did previously for the multiple-choice answers. For each question, we randomly selected one of the six human expert’s retrieval responses, comparing its correctness to the correctness of the given retrieval strategy’s response. This process was repeated 1,000,000 times given the large number (6 45 ) of possible combinations. Finally, we used two separate logistic regression Generalised Linear Models (GLMs) to test the null hypotheses that there were no differences in 1.) the accuracy of different LLMs at answering questions; and 2.) the retrieval accuracy of sparse, dense, and hybrid retrieval strategies. GLM selection was carried out via likelihood ratio tests of nested GLMs using the ‘anova’ function and Aikake Information Criterion corrected for small sample sizes (threshold of > 2 ΔAICc). Our initial GLM consisted of two interaction terms: LLM accuracy ~ Synopsis* LLM + LLM *Exam Type to test whether certain LLMs performed better for certain synopses, and whether certain LLMs performed better for certain types of exams (e.g., closed book, oracle, hybrid retrieval, etc.). The best GLM selected contained all the explanatory variables but without any interactions: LLM accuracy ~ Synopsis + LLM + Exam Type. GLMs for each possible combination of variables were also tested, including a null intercept-only GLM. For the second GLM, we initially started with an interaction term: Retrieval accuracy ~ Synopsis*Retrieval Strategy, but the best GLM selected contained both variables but without an interaction (Retrieval accuracy ~ Synopsis + Retrieval Strategy). GLMs for each possible combination of variables were also tested, including a null intercept-only GLM. Statistical significance of covariates in the best GLMs were assessed with an analysis of deviance test (Type II) using the ‘Anova’ function from the R package car (Fox and Weisberg 2019). We conducted pairwise tests between the levels of the categorical explanatory variables using the emmeans package with a Tukey adjustment for multiplicity (Lenth 2021). Results Comparison with human experts Open-book performance of LLMs demonstrates competitiveness with human expert performance Across all 45 selected questions, three LLMs, GPT-4o (97.8%), Llama 3.1 70B (97.8%), and Gemma 2 27B (95.6%) outperformed the average performance of human experts (mean = 94.8%; median = 95.6%; IQR = 93.8–95.6%; Range = 91.1%-97.8%) when provided access to the Conservation Evidence corpus and using a hybrid retrieval strategy (Table 1 ). All LLMs performed significantly better than a random guesser (means of 27.8–32.8 more correct answers versus a random guesser; p = 0 for all LLMs; Table S2), reflecting the fact that the LLM percentage accuracy was substantially higher than the 25% expected by chance. Most LLMs had a comparable level of performance at correctly answering questions to the average human (Table 1 ). Mean test statistic values were typically small with narrow 95% Confidence Intervals (CIs) that overlapped with zero, indicative of the large number of draws observed in our datasets (beyond those expected by chance) and thus the highly similar performance of each LLM and a randomly selected human across the 45 questions (Table 1 ). Therefore, for most LLMs, we could not reject the null hypothesis (p > 0.05; Table 1 ) that there was no difference in the head-to-head performance of the LLM and a randomly selected human expert, except for Llama 3.1 8B Instruct Turbo which performed significantly worse than the human experts (mean = 3.7 fewer correct answers; 95% CIs = 2–5 fewer correct answers; p = 0.008; Table 1 ). Gemma2–9B Instruct also appeared to perform worse than a random human expert (mean = 2.7 fewer correct answers; 95% CIs = 1–4 fewer correct answers; Table 1 ), but we could not reject the null hypothesis at the 0.05 significance level due to adjusting p-values for multiple comparisons. Conversely, GPT-4o and Llama 3.1 70B Instruct Turbo almost always matched or exceeded human experts (mean = 1.3 more correct answers; 95% CIs = 0–3 more correct answers; Table 1 ). Table 1 – Results of a paired comparison of question accuracy of Large Language Models using a hybrid retrieval strategy versus human experts, as well as comparing retrieval accuracy of different retrieval strategies versus human experts for the filtered 45-question set. Test statistics are from a permutation test used to test two null hypotheses: 1. LLM: no difference in multiple-choice question accuracy of the given LLM and a randomly selected human expert; 2. Retrieval strategies: no difference in retrieval accuracy of a retrieval strategy and a randomly selected human expert. The test statistic can be interpreted as follows: negative values = random guesser or randomly selected human expert answered more questions correctly; positive values = LLM/retrieval strategy answered more questions correctly, and zero = all draws or equal numbers of wins and losses. The test statistic accounts for the number of draws expected by chance for the multiple-choice questions. *denotes statistically significant Holm-adjusted p-value at 0.05 significance level. Large Language Model Overall percentage question accuracy Holm- adjusted p-value Mean permutation test statistic SD permutation test statistic Lower 95% CI Upper 95% CI GPT-4o 97.80% 1.000 1.331 1.000 0.000 3.000 Llama 3.1 70B Instruct Turbo 97.80% 1.000 1.335 1.000 0.000 3.000 Gemma2–27B Instruct 95.60% 1.000 0.333 0.999 -1.000 2.000 GPT-4o Mini 93.30% 1.000 -0.667 1.000 -2.000 1.000 Mixtral 8x22B 93.30% 1.000 -0.667 1.000 -2.000 1.000 Claude 3.5 Sonnet 93.30% 1.000 -0.665 1.001 -2.000 1.000 Gemini 1.5 Flash 91.10% 0.801 -1.667 1.000 -3.000 0.000 Gemini 1.5 Pro 91.10% 0.801 -1.668 1.000 -3.000 0.000 Gemma2–9B Instruct 88.90% 0.135 -2.666 1.000 -4.000 -1.000 Llama 3.1 8B Instruct Turbo 86.70% 0.008* -3.668 1.000 -5.000 -2.000 Retrieval strategy Overall percentage retrieval accuracy Holm- adjusted p-value Mean permutation test statistic SD permutation test statistic Lower 95% CI Upper 95% CI Dense 80.0% 0.029* -3.500 1.404 -6.000 -1.000 Sparse 71.1% 0.000* -7.501 1.407 -10.000 -5.000 Hybrid 88.9% 0.270 0.500 1.404 -2.000 3.000 Comparison of retrieval strategy performance with expert retrieval The hybrid strategy’s overall retrieval accuracy (i.e., identifying the correct source document to answer the question) was 88.9%, narrowly outperforming the average of human experts (mean = 87.8%; median = 88.9%; IQR = 85.6–88.9%; Range = 82.2–93.3%). Using equivalent permutation tests as before, we found that the per-question retrieval accuracy of dense and sparse retrieval strategies was significantly worse than a randomly selected human expert (Table 1 – Dense: mean = 3.5 fewer correct retrievals, 95% CIs = 1–6 fewer correct retrievals, p = 0.029; Sparse: mean = 7.5 fewer correct retrievals, 95% CIs = 5–10 fewer correct retrievals, p < 0.001). However, we could not reject the null hypothesis (of no difference in retrieval accuracy versus a random expert) for the hybrid retrieval strategy for which we found comparable levels of accuracy with humans (mean = 0.5 more correct retrievals, 95% CIs = 2 fewer to 3 more correct retrievals; p = 0.271; Table 1 ). This was again associated with large numbers of draws between human experts and the hybrid retrieval strategy. The median time taken to answer each question by a human expert was 139.5 seconds (IQR 90.9-269.1 seconds) versus practically instantaneous responses from any LLM (given retrieval had already been completed and documents effectively cached). Model comparisons across unfiltered dataset Question accuracy across synopses, exam conditions, and retrieval strategies We found that for the larger dataset of questions, LLM accuracy differed significantly between synopses, LLMs, and exam conditions (Table S3, S4). The accuracy of LLMs was significantly higher for the Biodiversity of Marine Artificial Structures and Natural Pest Control synopses than almost all other synopses – differences were non-significant with each other and the Bat, Bee, and Natural Pest Control synopses (Table S5, Fig. 1 ). Synopses that LLMs performed worst for were Reptile, Control of Freshwater Invasives, and Butterfly and Moth Conservation (Fig. 1 ; Table S4, S5). Generally, LLMs had significantly higher accuracy under oracle conditions than all other exam types, followed by confused, hybrid retrieval, dense retrieval, sparse retrieval, and closed book in descending order – all differences were statistically significant (Table S6). GPT-4o had significantly higher accuracy than all other LLMs - although the difference was non-significant for Llama 3.1 70B Instruct Turbo (OR = 1.118, z = 2.807, p = 0.134; Table S7). Llama 3.1 8B had significantly lower accuracy than all LLMs, whilst next worst was Mixtral 8x22 which had significantly lower accuracy than the rest of the LLMs (Table S7). LLMs exhibited lower performance in closed-book conditions compared to open-book scenarios (Fig. 1 ; Figure S3; Table 2 ). When presented with only the correct document all LLMs showed high levels of performance (Table 2 ) with some, such as GPT-4o and Llama 3.1-70B reaching 100% answer correctness across the human filtered questions. Performance declined across all LLMs when extraneous, irrelevant documents were introduced in the 'confused' scenario (Figure S3; Table 2 ). Table 2 – Overall Large Language Model (LLM) accuracy across different exam conditions on the unfiltered dataset. The table is sorted by LLM performance under the hybrid retrieval strategy. Results for the filtered 45-question dataset are presented in Table S8. We also specify whether LLMs are open or closed source. LLM Closed Book Oracle Confused Dense Sparse Hybrid Open/closed source GPT-4o 69.8% 95.3% 94.4% 86.1% 83.2% 90.2% Closed Llama 3.1 70B Instruct Turbo 68.7% 94.3% 92.7% 85.3% 82.5% 87.9% Open Gemini 1.5 Pro 65.5% 94.9% 92.8% 84.4% 81.3% 87.9% Closed Gemma2–27B Instruct 66.1% 93.7% 92.3% 83.6% 80.7% 87.2% Open GPT-4o Mini 66.5% 92.4% 91.3% 83.4% 81.1% 86.5% Closed Gemma2–9B Instruct 63.2% 92.5% 91.4% 81.7% 79.0% 85.9% Open Gemini 1.5 Flash 64.3% 93.4% 91.7% 82.5% 80.0% 85.9% Closed Claude 3.5 Sonnet 65.4% 93.4% 91.4% 80.7% 78.1% 83.9% Closed Mixtral 8x22B 64.3% 91.2% 90.1% 79.2% 75.2% 83.0% Open Llama 3.1 8B Instruct Turbo 62.6% 90.4% 86.1% 77.8% 73.9% 80.1% Open Retrieval accuracy across synopses, exam conditions, and retrieval strategies Retrieval accuracy significantly differed across both synopses and retrieval strategies (Fig. 2 ; Table S9, S10). Across all LLMs, hybrid retrieval accuracy was significantly better than both Dense and Sparse retrieval (Table 3 ; Table S10, S11), as overall there was an increase in retrieving the correct document from 61.8% and 75.2–83.2% from Sparse and Dense to Hybrid, respectively. This aligns with the increased accuracy of LLMs with hybrid retrieval strategy compared to LLMs with dense or sparse retrieval strategies (Fig. 1 ; Fig. S3; Table S6). Table 3 – Overall retrieval accuracy of different retrieval strategies across the unfiltered dataset of questions. Retrieval strategy Percentage of questions where source action document retrieved correctly Percentage of questions where document from same synopsis was retrieved (considers both selected documents) Percentage of questions where source action document was selected first (if one of two documents selected was correct) Dense 75.2% 85.8% 80.0% Sparse 61.8% 62.7% 82.6% Hybrid 83.2% 84.7% 78.2% Across all synopses, hybrid retrieval accuracy was significantly greater than both dense (OR = 1.66, z = 6.17, p < 0.0001; Table S11) and sparse retrieval (OR = 3.14, z = 14.5, p < 0.0001; Table S11), as was found for LLM accuracy using dense versus sparse retrieval (Fig. 1 ; Table S6). Dense retrieval accuracy was also significantly greater than sparse retrieval (OR = 1.89, z = 8.75, p < 0.0001; Table S11). Dense retrieval resulted in the correct (as used to generate the question) document for 75.2% of questions whilst this was only 61.8% for sparse retrieval (Table 3 ). Notably, in 85.8% of questions, dense retrieval returned at least one document from the same synopsis of the source document, for Sparse this was 62.7% - only marginally higher than returning the correct document. For synopses, the poorest retrieval accuracy was found for the Mediterranean Farmland synopsis and significantly lower than more than half of the synopses (Fig. 2 ; Table S9). Several synopses (e.g., Marine and Freshwater Mammals, Marine Artificial Structures, Subtidal Benthic Invertebrates) had significantly higher retrieval accuracy than some of the synopses with the lowest retrieval accuracies (e.g., Reptile Conservation and Mediterranean Farmland; Fig. 2 ; Table S12). Generational Progress To assess the improvement of cutting-edge LLMs over time, we tested GPT-3.5 Turbo Instruct (released in September 2023 but derived from GPT-3.5 originally released in November 2022) on 679 unfiltered questions (Table S13). Performance in the Oracle scenario was notably low for GPT-3.5 Turbo Instruct compared to all ten, more recent LLMs (Fig. 1 ; Table 1 ). Discussion Our findings that most LLMs tested using a hybrid retrieval strategy, in particular GPT-4o and Llama 3.1 70B Instruct Turbo, demonstrated comparable performance at answering multiple choice questions with a randomly selected human expert and far exceeded performance expected from random guessing. In addition, retrieval accuracy for the hybrid strategy was also competitive with human expert retrieval. The retrieval performance was found to be lower than answer correctness for both LLMs and humans, likely due to overlapping key messages between actions, which enables related actions to inform responses to other questions. An example of this phenomenon is provided in Figure S1. Our findings also suggest that the closed-book performance of LLMs (without any retrieval strategy) demonstrated a level of conservation-specific knowledge, as evidenced by the fact that all LLMs performed better than random guessing under all exam conditions. This varied by topic area (Conservation Evidence synopses) but results remained high enough to suggest that the pre-training corpora incorporated relevant conservation literature – whilst many studies are published behind paywalls, the Conservation Evidence database website ( https://www.conservationevidence.com ) and synopses are freely available online. We also found that the performance of LLMs was consistent between the following exam conditions: oracle (providing the correct source action document) and confused (providing both a correct source action document with a random one). Since retrieval always results in two documents being provided to the LLM, the confused exam condition sets a ceiling for their performance – as expected, all LLMs have lower open-book performance (using hybrid, dense, or sparse retrieval) than in oracle or confused performance across the larger unfiltered dataset. The hybrid retrieval strategy was found to perform substantially better than both dense and sparse retrieval strategies in terms of both question and retrieval accuracy, with dense also outperforming sparse. We also found that whilst the dense retrieval strategy’s performance was higher when only considering whether it returned at least one document from the same synopsis of the source document, there was negligible improvement for the sparse retrieval strategy. This is likely attributable to actions within the same synopsis being semantically similar and dense embeddings being able to capture this. Finally, our findings also suggest there has been substantial generational improvement in LLM performance over a period of approximately two years, given that GPT-3.5 Turbo Instruct (derived from GPT-3.5 released in 2022) performed notably poorer than all ten more recently developed LLMs that we tested. This includes against models such as GPT-4o mini and Gemini 1.5 Flash which are considerably smaller and cheaper. Therefore, there does appear to be the potential for further improvements to the models we tested over the next few years, and emphasises the importance of our validation pipeline approach to rapidly test new models as they are released and calibrate them against expert-level training data. Limitations and future work These findings need to be considered in the context of certain limitations. First, our results relate to LLMs’ specific performance on the Conservation Evidence database, which may not generalise to questions on other conservation questions. We also may not have captured the full range of questions that decision-makers have as the questions were derived from the Conservation Evidence database and generated by a LLM - thus any gaps and biases in the CE database and the LLM’s pre-training data, including geographic, taxonomic, and language bias 31 – 33 may influence the questions set and thus the observed performance of LLMs. Nevertheless, we found that the observed performance of LLMs was broadly similar between the human-filtered questions and the larger dataset of unfiltered questions. However, LLMs are ultimately black box models, and we have a poor understanding of the pre-training data used to build them and their associated biases – thus further testing across different subject areas and domains remains important. Our results suggest that performance for certain synopses was poorer than others, although this did not necessarily follow an obvious taxonomic or biogeographical pattern. Our approach could be used to test LLM performance when retrieving information from databases, repositories and syntheses that already exist in other disciplines and topic areas – e.g., web databases such as the IUCN Red List and Education Endowment Foundation Toolkits that could be treated in a similar way to how the Conservation Evidence database was in our study. It is also important to note that our evaluation method used Claude 3.5 Sonnet to generate the questions and answers and so all our questions should have been framed in a manner answerable by LLMs. However, Claude 3.5 Sonnet only performed in the middle to lower end of other LLMs tested, suggesting that it received no significant advantage in answering questions over the others. We must also acknowledge that despite setting the hyperparameter ‘temperature’ of all LLMs to zero, we cannot completely guarantee that LLM responses were 100% deterministic. However, whilst it is possible that if prompted a large number of times, LLMs may have provided a different response to a small number of questions, this is unlikely to have altered our major findings. The human expert group we used was also limited in its sample size because we focused on testing LLMs against humans that were highly familiar with navigating and answering questions using the Conservation Evidence database. However, this means that the human expert group we used is likely to represent an extremely high threshold of human performance compared to that of a more diverse group of experts with conservation expertise but with less expertise on using the database or synthesising evidence on conservation interventions. In this context, however, the LLM performance we observed is even more impressive. We may have also underestimated the retrieval times for human experts given that the questions we generated were based on a single conservation action and thus had a relatively high level of specificity – it may have taken experts longer to answer questions that rely on synthesising information across multiple conservation actions, and certainly would have taken a wider group of conservation experts without specific knowledge of the database more time too. Therefore, there are considerable potential time savings to be achieved by querying the Conservation Evidence database via a LLM-based RAG system over asking a human expert. Finally, our exam-style evaluation scheme was designed to provide a plausible upper bound on LLMs performance at answering questions as a first step to evaluating their suitability for decision support when drawing upon evidence databases and syntheses. There is likely to be more potential for errors when answering questions using free text generation as opposed to the constrained multiple-choice answers we examined. Furthermore, our evaluation only included a single task: answering a multiple-choice question on a single conservation action from a single synopsis. Now that we have demonstrated LLMs using a hybrid retrieval strategy are competitive with human experts at simple retrieval and question answering tasks, future research should investigate at which point LLM performance declines as tasks become more complex in terms of both questions and responses (e.g., questions requiring more synthesis across multiple actions or synopses with nuanced answers). Once this performance threshold with task complexity is identified, future systems should then be designed to stay above that threshold to avoid the risk of misinforming decision-makers. However, there is also the potential to test whether more advanced prompt engineering can improve the performance of LLMs, as well as more computationally expensive embedding approaches that might improve retrieval performance. Risks It is also important to consider the ethics and clear associated risks of using LLMs in a decision support capacity to answer conservation questions based on evidence databases. First, equitable access to using LLMs for these purposes is important – any implementation should ideally be free and open access. This may also place an important constraint on which LLMs can be used in terms of cost – therefore, it is important to consider the relative cost-effectiveness of LLMs by comparing their performance to their cost. Indeed, we found that many open-source LLMs performed just as well as closed source LLMs (Table 2 ). Furthermore, access to LLMs will always be constrained, at least in part, by access to technology, power, and stable internet connections – although most decision support tools should be able to be run on relatively cheap devices, such as mobile phones. Therefore, the consideration of the power dynamics between Global North and Global South institutions and organisations owning and using these tools should be carefully considered and recognised to avoid unequitable relationships at the research-practice and research-policy interfaces 34 . It is also important to recognise that the training of LLMs comes with substantial embodied environmental costs. These environmental costs have already occurred prior to the release of these models (i.e., in their initial training) and so there is arguably an ethical case that once developed LLMs should be used as much as possible for tasks that might mitigate these environmental costs. However, there are still costs associated with LLM inference via APIs (although these are a fraction of the original training costs) and so efficiency in the design of LLM-based RAG systems needs to be prioritised – thus we should make sure that if there is a choice between using models with similar levels of performance, that we select the simpler, computationally cheaper models with lower environmental costs whenever possible. There is also a risk that LLMs may lead to deskilling of conservation practitioners and scientists in searching for, and assessing, the evidence. Although the goal is to make accessing and interpreting evidence databases and syntheses easier and more efficient, there is the risk that practitioners and scientists may think less critically about the evidence, its limitations, source, and validity. This could also lead to a lack of accountability in the use of evidence, including biased evidence use to support self-serving lines of argument and political rhetoric (e.g., by organisations for greenwashing). Therefore, it will be important to carefully design AI-assisted decision support systems to ensure they prompt users to understand the uncertainty and limitations associated with the evidence base, including its reliability and local relevance and transferability 5 , 16 , 31 , 35 , 36 . Conclusion We have shown that LLMs have comparable performance as human experts at providing restricted responses and retrieving relevant information to conservation intervention questions using the Conservation Evidence database. This first step in evaluating the performance of LLMs at decision support tasks now enables research to move on to establishing where any performance issues may occur when LLMs answer more complex, nuanced questions with open-ended, free-text responses. We stress that if performance issues are observed, we should design systems accordingly above this threshold to ensure the risk of misinforming decision-makers is minimised (with comparable or lower levels of error to human experts). For the Conservation Evidence database, our current findings suggest that it is justifiable to implement an LLM capable of providing an intelligent search function, to help direct users of the website to the most suitable page where evidence exists to answer their question. Our findings currently suggest that general LLMs used ‘out-of-the-box’ are likely to perform poorly at giving evidence-based advice or recommendations for decision-making. Any decisions based on such information are likely to be misinformed. Therefore, we urge those thinking of using general LLMs ‘as-is’ to provide decision support within their organisation or to end users for certain tasks, that they spend time and effort to carefully consider how to design and evaluate LLM-based systems using retrieval augmented generation (RAG) and a hybrid retrieval strategy. Looking to the future, if the rapid pace of improvements to underlying models continue, it is possible that better performing LLMs coupled with further refinements of prompts and retrieval strategies can enable the development of RAG systems capable of expert-level, evidence-based advice based on evidence databases and syntheses. However, careful and considered evaluation of LLM-based decision support systems is needed along the way to ensure that more rapid and intuitive access to relevant evidence to inform practice and policy does not come at the cost of misinforming decision-makers with biased, erroneous, or misleading information. Declarations Conflict of interest disclosure: The authors have no conflicts of interest to disclose. Funding statement: RI was supported by a UROP internship at the University of Cambridge. APC received financial support from Imperial College London through an Imperial College Research Fellowship grant, as well as a Henslow Fellowship funded by the Cambridge Philosophical Society. Authors contributions: RI: Formal Analysis, Data Curation, Investigation, Methodology, Software, Validation, Writing – Original Draft Preparation, Writing – Review & Editing. Data availability statement: Data and code associated with this paper can be found at: https://doi.org/10.5281/zenodo.14049626 . References Leclère D et al (2020) Bending the curve of terrestrial biodiversity needs an integrated strategy. Nature 585:551–556 Sutherland WJ, Pullin AS, Dolman PM, Knight TM (2004) The need for evidence-based conservation. Trends Ecol Evol 19:305–308 Sutherland WJ, Wordley CF (2017) R. Evidence complacency hampers conservation. Nat Ecol Evol 1:1215–1216 Walsh JC, Dicks LV, Sutherland WJ (2015) The effect of scientific evidence on conservation practitioners’ management decisions. Conserv Biol 29:88–98 MacLeod CJ, Brandt AJ, Dicks LV (2022) Facilitating the wise use of experts and evidence to inform local environmental decisions. People Nat 4:904–917 Cook CN, Nichols SJ, Webb JA, Fuller RA, Richards RM (2017) Simplifying the selection of evidence synthesis methods to inform environmental decisions: A guide for decision makers and scientists. Biol Conserv 213:135–145 White H (2019) The twenty-first century experimenting society: the four waves of the evidence revolution. Palgrave Commun 5:47 Cooke SJ et al (2023) Environmental evidence in action: on the science and practice of evidence synthesis and evidence-based decision-making. Environ Evid 12:10 Kadykalo AN et al (2021) Bridging research and practice in conservation. Conserv Biol 35:1725–1737 Walsh JC, Dicks LV, Raymond CM, Sutherland WJ (2019) A typology of barriers and enablers of scientific evidence use in conservation practice. J Environ Manage 250:109481 Rose DC et al (2018) The major barriers to evidence-informed conservation policy and possible solutions. Conserv Lett 11 Hager P et al (2024) Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med 30:2613–2622 Sandmann S, Riepenhausen S, Plagwitz L, Varghese J (2024) Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks. Nat Commun 15:2050 Scheepens D, Millard J, Farrell M, Newbold T (2024) Large language models help facilitate the automated synthesis of information on potential pest controllers. Methods Ecol Evol 15:1261–1273 Spillias S et al (2024) Human-AI collaboration to identify literature for evidence synthesis. Cell Rep Sustain 1:100132 Berger-Tal O et al (2024) Leveraging AI to improve evidence synthesis in conservation. Trends Ecol Evol 39:548–557 Sworna ZT, Urzedo D, Hoskins AJ, Robinson CJ (2024) The ethical implications of Chatbot developments for conservation expertise. AI Ethics. 10.1007/s43681-024-00460-3 Kaiser KN et al (2024) Accuracy and consistency of publicly available Large Language Models as clinical decision support tools for the management of colon cancer. J Surg Oncol. 10.1002/jso.27821 Weidinger L et al (2022) Taxonomy of Risks posed by Language Models. in. ACM Conference on Fairness, Accountability, and Transparency 214–229 (ACM, New York, NY, USA, 2022). 10.1145/3531146.3533088 De Angelis L et al (2023) ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front Public Health 11 Au Yeung J et al (2023) AI chatbots not yet ready for clinical use. Front Digit Health 5 Schwartz IS, Link KE, Daneshjou R (2024) Cortés-Penfield, N. Black Box Warning: Large Language Models and the Future of Infectious Diseases Consultation. Clin Infect Dis 78:860–866 Shen Y et al (2023) ChatGPT and Other Large Language Models Are Double-edged Swords. Radiology 307 Martin TG et al (2012) Acting fast helps avoid extinction. Conserv Lett 5:274–280 Leyer M, Schneider S (2021) Decision augmentation and automation with artificial intelligence: Threat or opportunity for managers? Bus Horiz 64:711–724 Adam H, Balagopalan A, Alsentzer E, Christia F, Ghassemi M (2022) Mitigating the impact of biased artificial intelligence in emergency decision-making. Commun Med 2:149 Augenstein I et al (2023) Factuality Challenges in the Era of Large Language Models Salemi A, Kallumadi S, Zamani H (2024) Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation. in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval 752–762ACM, New York, NY, USA. 10.1145/3626772.3657783 Quigley K, Koker T, Taylor J, Mancuso V, Brattain L (2024) Designing Retrieval-Augmented Language Models for Clinical Decision Support. 159–171. 10.1007/978-3-031-63592-2_13 Guinet G, Omidvar-Tehrani B, Deoras A, Callot L (2024) Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation Christie AP et al (2020) Poor availability of context-specific evidence hampers decision-making in conservation. Biol Conserv 248 Christie AP et al (2020) The challenge of biased evidence in conservation. Conserv Biol. 10.1111/cobi.13577 Amano T et al (2021) Tapping into non-English-language science for the conservation of global biodiversity. PLoS Biol 19:e3001296 Miller J, White TB, Christie AP (2023) Parachute conservation: Investigating trends in international research. Conserv Lett 16 Christie AP et al (2019) Simple study designs in ecology produce inaccurate estimates of biodiversity responses. J Appl Ecol 56:2742–2754 Christie AP et al (2020) Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences. Nat Commun 11:6377 Additional Declarations The authors declare no competing interests. Supplementary Files ManuscriptPREPRINTV1.2SuppInfo.docx Supporting Information Cite Share Download PDF Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5409185","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":377506039,"identity":"6478e3a8-11e6-4a65-ba93-006a93fc9d05","order_by":0,"name":"Radhika Iyer","email":"","orcid":"","institution":"Department of Computer Science \u0026 Technology, University of Cambridge","correspondingAuthor":false,"prefix":"","firstName":"Radhika","middleName":"","lastName":"Iyer","suffix":""},{"id":377506038,"identity":"ee9fe857-32a1-41a0-8670-2db1e73dd7f2","order_by":1,"name":"Alec Christie","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAvUlEQVRIiWNgGAWjYHAC9g8fKuCcBKK0sDHOOEOqFmbeNlK08M9ufvZw5jw7ed0G5ocfGNvSCGuRuHPM3ODjtmTDbQfYjCUY23KIcNaNHAbJmdsOMG47wGDGwNhWQViHPFCLNO+cA/bbDrB/I06LwY0cNmnehgOJ2w7wgGwhwmGGd44ZG844lpy87TBPsUTCOSK8L3e7+eGDDzV2ttuOt2/88KEsmbAWBgkYg5mB2IiUIKxkFIyCUTAKRjoAAJ1IOlVV1B1KAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-8465-8410","institution":"Imperial College London","correspondingAuthor":true,"prefix":"","firstName":"Alec","middleName":"","lastName":"Christie","suffix":""},{"id":377506040,"identity":"77c76640-eebb-4a1e-830c-db6a225415df","order_by":2,"name":"Anil Madhavapeddy","email":"","orcid":"","institution":"Department of Computer Science \u0026 Technology, University of Cambridge","correspondingAuthor":false,"prefix":"","firstName":"Anil","middleName":"","lastName":"Madhavapeddy","suffix":""},{"id":377506041,"identity":"1a6be47f-15fb-41b9-b3e5-235e371797c7","order_by":3,"name":"Sam Reynolds","email":"","orcid":"https://orcid.org/0000-0002-1255-580X","institution":"University of Cambridge","correspondingAuthor":false,"prefix":"","firstName":"Sam","middleName":"","lastName":"Reynolds","suffix":""},{"id":377506042,"identity":"d5d5fa41-3fe0-4830-9619-dda5113c47d9","order_by":4,"name":"William Sutherland","email":"","orcid":"","institution":"University of Cambridge","correspondingAuthor":false,"prefix":"","firstName":"William","middleName":"","lastName":"Sutherland","suffix":""},{"id":377506043,"identity":"a90c903a-0af9-4e7a-92dd-092efa86363d","order_by":5,"name":"Sadiq Jaffer","email":"","orcid":"","institution":"Department of Computer Science \u0026 Technology, University of Cambridge","correspondingAuthor":false,"prefix":"","firstName":"Sadiq","middleName":"","lastName":"Jaffer","suffix":""}],"badges":[],"createdAt":"2024-11-07 10:46:47","currentVersionCode":2,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-5409185/v2","doiUrl":"https://doi.org/10.21203/rs.3.rs-5409185/v2","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":74577392,"identity":"4f17a5a7-b55b-4595-82e6-a885ee94e3ce","added_by":"auto","created_at":"2025-01-23 15:14:42","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":214307,"visible":true,"origin":"","legend":"\u003cp\u003eLogistic regression Generalised Linear Model (GLM) predictions of the accuracy of LLMs across different synopses, under different exam types (mean and 95% Confidence Intervals). The results for confused, sparse, and dense retrieval are found in Figure S3.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-5409185/v2/0771a82b0d68d5e55611fa00.png"},{"id":74576227,"identity":"8050311e-addc-47e2-ae0e-0b6905d83fe6","added_by":"auto","created_at":"2025-01-23 15:06:42","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":104619,"visible":true,"origin":"","legend":"\u003cp\u003eLogistic regression Generalised Linear Model (GLM) predictions of the retrieval accuracy of different retrieval strategies across different synopses (mean and 95% Confidence Intervals).\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-5409185/v2/e3c7ede9c5f85a8e0464c5ba.png"},{"id":74578685,"identity":"58437527-f742-4049-b3c5-c10b0a10da05","added_by":"auto","created_at":"2025-01-23 15:30:43","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1384155,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5409185/v2/b9c5cf74-9b77-4d2d-8c77-5f39d6395ca8.pdf"},{"id":74577391,"identity":"9b2045dd-43f5-4682-8680-bcb0eac06313","added_by":"auto","created_at":"2025-01-23 15:14:42","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":482391,"visible":true,"origin":"","legend":"\u003cp\u003eSupporting Information\u003c/p\u003e","description":"","filename":"ManuscriptPREPRINTV1.2SuppInfo.docx","url":"https://assets-eu.researchsquare.com/files/rs-5409185/v2/5bc970f96d8ce6aa94877c46.docx"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eCareful design of Large Language Model pipelines enables expert-level retrieval of evidence-based information from conservation syntheses\u003c/p\u003e","fulltext":[{"header":"Introduction","content":"\u003cp\u003eTo maximise our chances of bending the curve of biodiversity loss\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u003c/sup\u003e with limited time and resources, we need to use the best available evidence to increase the effectiveness of conservation efforts whilst also reducing harmful impacts to both nature and local communities\u003csup\u003e\u003cspan additionalcitationids=\"CR3 CR4\" citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e–\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e. Whilst evidence syntheses are key to informing more effective practice and policy by distilling the scientific literature\u003csup\u003e\u003cspan additionalcitationids=\"CR7\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e–\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e, they are not always easy for decision-makers to access, digest and use\u003csup\u003e\u003cspan additionalcitationids=\"CR10\" citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e–\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e. However, recent advances in the capabilities of Large Language Models (LLMs; Box 1) have made it easier than ever to gain access to summaries of information and on-demand answers to specific questions across a variety of domains\u003csup\u003e\u003cspan additionalcitationids=\"CR13 CR14\" citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e–\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e. LLMs clearly have considerable potential to summarise information efficiently from vast corpuses of literature, but their use in rigorously supporting effective decision-making remains \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"controversial\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003econtroversial\u003c/em\u003e and fraught with potential pitfalls and dangers, including the risk of misinformation from biases and errors\u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e,\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eThere is particular concern that the proliferation of natural language interfaces or ‘chatbots’ (e.g., OpenAI’s \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"chatgpt\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eChatGPT\u003c/em\u003e, Microsoft Co-Pilot, and Google’s Gemini) could misinform decision-makers who may be attracted to LLMs for their ability to rapidly and flexibly answer their questions\u003csup\u003e\u003cspan additionalcitationids=\"CR19 CR20 CR21 CR22\" citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e–\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e, particularly in crisis disciplines such as biodiversity conservation\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e. \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e usage by those providing advice to decision-makers (e.g., consultants and scientists) could also further compound poor quality decision-making without properly considering the quality and relevance of evidence\u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e,\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e,\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e,\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u003c/sup\u003e. For example, LLMs used ‘out-of-the-box’ have the potential for hallucinations, false references and citations, out-of-date and biased information based on inherent biases in the training data used to build these models\u003csup\u003e\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e. It is therefore vital to determine whether reliable implementations of LLMs that derive their information from robust evidence syntheses and databases can be developed that enable rapid access to evidence bases without misinforming decision-makers.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003ctable float=\"No\" id=\"Taba\" border=\"1\"\u003e\u003ccolgroup cols=\"1\"\u003e\u003c/colgroup\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eBox 1 – Glossary and explanation of study data source and methodology\u003c/b\u003e\u003c/p\u003e \u003cp\u003e\u003cb\u003eLarge Language Models\u003c/b\u003e (LLMs) are neural networks trained on large quantities of text. Given a block of text, called the \u003cem\u003econtext\u003c/em\u003e, they are optimised to predict the small piece of text that follows. Adding this small piece of text to the context and repeating the process enables text generation. With tuning these models can follow instructions and carry out multi-turn conversations, exemplified in \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"chatgpt\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eChatGPT\u003c/em\u003e.\u003c/p\u003e \u003cp\u003e\u003cb\u003eContext\u003c/b\u003e is the initial set of text given to a \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e that contains instructions and potentially data for it to use. In the Transformer model used by all state-of-the-art models (and all tested in this paper) the computational resources required scales quadratically with context window. This makes responses to long contexts expensive and is the motivation behind techniques for only including relevant information in the context.\u003c/p\u003e \u003cp\u003e\u003cb\u003eRetrieval-Augmented Generation\u003c/b\u003e (RAG) is one such technique for identifying relevant information to populate the context. In this approach the user query is used to \u003cem\u003eretrieve\u003c/em\u003e potentially relevant documents from a corpus, and these are then added to the context along with the user query and sent to the \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e.\u003c/p\u003e \u003cp\u003eThe \u003cb\u003eConservation Evidence \u003cem class=\"Highlight ht4fc55b9d-f515-4fa8-9e5d-6731d62f45ba\" highlight=\"true\" htmatch=\"database\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003edatabase\u003c/em\u003e\u003c/b\u003e (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e\u003ca href=\"http://www.conservationevidence.com\" target=\"_blank\"\u003ewww.conservationevidence.com\u003c/a\u003e\u003c/span\u003e\u003cspan address=\"http://www.conservationevidence.com\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e) contains over 8800 studies, at time of writing, testing the effectiveness of over 3800 conservation interventions on biodiversity outcomes (of which 2,250 have studies testing them). It is structured into topic areas called synopses (e.g., Amphibian, Peatland, and Bird Conservation), within which study summaries are organised under the different actions they test. For each action, key message paragraphs summarise the findings of studies and background paragraphs provide key contextual information describing the conservation action being considered. Each action is then assessed using structured expert elicitation to provide an effectiveness category that accounts for the potential effectiveness and harms of the action and the certainty of the evidence.\u003c/p\u003e \u003cp\u003eA \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e (Claude 3.5 Sonnet) was used to generate a multiple-choice exam for each of the 2,250 actions using an automated method\u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e, (Panel A and B in the figure below) excluding questions that solely asked questions based on the effectiveness categories in the Conservation Evidence databases. This formed a larger set of 1867 unfiltered questions. We also refined these down to a filtered set of 45 questions to enable a comparison with \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehuman\u003c/em\u003e experts, ensuring that filtered questions were clear and could be answered with a single, accurate answer. Ten LLMs were then asked to provide answers under six different exam conditions for each question within the unfiltered and filtered sets (Panel B in figure below). These exam conditions included three different types of retrieval strategies: sparse, dense, and hybrid retrieval.\u003c/p\u003e \u003cp\u003e\u003cb\u003eSparse retrieval\u003c/b\u003e uses keyword-based metrics to find relevant documents from amongst a corpus. This can work well when there is significant keyword overlap between queries and documents but can fail when semantically related documents do not share keywords.\u003c/p\u003e \u003cp\u003e\u003cb\u003eDense retrieval\u003c/b\u003e uses a small language model to \u003cem\u003eembed\u003c/em\u003e documents or smaller chunks of them as a point in a high dimensional space. These models are optimised to minimise the distance of semantically related text in the embedding space. The user query is embedded and the closest documents in the corpus are returned as relevant.\u003c/p\u003e \u003cp\u003e\u003cb\u003eHybrid retrieval\u003c/b\u003e uses a small language model called a \u003cem\u003ereranker\u003c/em\u003e to compare documents retrieved from other methods with the query. This can increase performance as the reranking model has access to the full text of potentially relevant documents.\u003c/p\u003e \u003cp\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003eFew studies have quantitatively evaluated the reliability of using LLMs for decision support\u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e,\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e and none, to our knowledge, in conservation have made comparisons with \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehuman\u003c/em\u003e experts. It is also unclear how recent advances in fine-tuning and text retrieval techniques (e.g., Retrieval Augmented Generation: RAG; Box 1) to improve the performance of LLMs in specific contexts and reduce errors may help to minimise the risks of misinforming decision-makers\u003csup\u003e\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e,\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e\u003c/sup\u003e. We also lack an understanding of the subject-specific information that different LLMs possess across different topic areas and therefore whether certain LLMs might provide more reliable decision support for certain fields.\u003c/p\u003e \u003cp\u003eBefore evaluating how LLMs perform at providing open-ended, free-text responses to questions, we first need to understand their performance at providing restricted responses to questions and retrieving the correct information. We expect \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e performance to decline as tasks become more complex and so we first need to establish where performance issues occur. Once a performance threshold is determined, we can design systems accordingly to ensure the risk of misinforming decision-makers is minimised, with comparable or lower levels of error than \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehumans.\u003c/em\u003e Here we begin this process by first testing \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e performance on relatively simple tasks with restricted responses.\u003c/p\u003e \u003cp\u003eWe ask ten different state-of-the-art LLMs to provide answers to conservation-related multiple-choice questions, including identifying the relevant source document used to answer each question. We use the Conservation Evidence \u003cem class=\"Highlight ht4fc55b9d-f515-4fa8-9e5d-6731d62f45ba\" highlight=\"true\" htmatch=\"database\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003edatabase\u003c/em\u003e as a \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"case study\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ecase study\u003c/em\u003e, asking LLMs questions on the effects of conservation interventions for which evidence exists in this \u003cem class=\"Highlight ht4fc55b9d-f515-4fa8-9e5d-6731d62f45ba\" highlight=\"true\" htmatch=\"database\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003edatabase\u003c/em\u003e (Box 1). We adapt and apply an automated method\u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e to measure the task-specific accuracy of Retrieval-Augmented LLMs using automatically generated synthetic exams composed of multiple-choice questions based on a given corpus of documents (Box 1; Methods). This represents a cost-efficient, interpretable, and robust strategy to select optimal components for an initial RAG system providing an intelligent search function for users of the Conservation Evidence \u003cem class=\"Highlight ht4fc55b9d-f515-4fa8-9e5d-6731d62f45ba\" highlight=\"true\" htmatch=\"database\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003edatabase\u003c/em\u003e. To determine its robustness to errors, we ask whether each \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e performs better at answering questions and retrieving the correct source document to answer questions than: 1. random guessing; 2. \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehuman\u003c/em\u003e experts in conservation evidence synthesis; 3. other LLMs; and 4. a predecessor of one of the LLMs - i.e., has there been improvement over time? Our approach could be applied across fields and disciplines to evaluate the suitability of different LLMs and retrieval strategies for providing decision support on specific subjects.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e\u003c/p\u003e "},{"header":"Methods","content":"\u003cp\u003eWe evaluated the performance of different Large Language Models (LLMs) in answering questions based on the evidence contained within the Conservation Evidence \u003cem class=\"Highlight ht4fc55b9d-f515-4fa8-9e5d-6731d62f45ba\" highlight=\"true\" htmatch=\"database\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003edatabase\u003c/em\u003e as a \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"case study\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ecase study\u003c/em\u003e. Our methodology was adapted from a previously published automated approach\u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e, which is designed to measure the task-specific accuracy of Retrieval-Augmented Generation (RAG) LLMs using automatically generated synthetic exams composed of multiple-choice questions based on a given corpus of documents (Box 1).\u003c/p\u003e\u003cp\u003eConservation Evidence Corpus\u003c/p\u003e\u003cp\u003eThe Conservation Evidence \u003cem class=\"Highlight ht4fc55b9d-f515-4fa8-9e5d-6731d62f45ba\" highlight=\"true\" htmatch=\"database\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003edatabase\u003c/em\u003e comprises a collation of studies testing the effects of conservation interventions on biodiversity outcomes for 26 topic areas (called synopses) structured around various species and habitat groups (e.g., Birds, Bees, Terrestrial Mammals). Each synopsis contains a range of actions (22 to 400 per synopsis), with each action including an action title and number, background information, key messages, and evidence summaries – this text constituted a ‘document’ for the purposes of our study (Fig. S1), if any evidence has been found to test that action. For this study, we focused on the 2,250 actions that had at least one associated study. See Figures S1 and S2 for examples of summaries of the evidence for different actions.\u003c/p\u003e\u003cp\u003eExam Generation\u003c/p\u003e\u003cp\u003eWe used Claude-3.5 Sonnet for exam question generation, as it was the strongest \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"publicly available\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003epublicly available\u003c/em\u003e \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e at the time of the study. The exam generation process involved several steps:\u003c/p\u003e\u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eCorpus Preparation\u003c/b\u003e: We reduced the action documents to key messages and removed actions without supporting evidence.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eQuestion Generation\u003c/b\u003e: We generated 3–4 questions per action using Claude-3.5 Sonnet (see Table S1 for prompt).\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eFiltering Process\u003c/b\u003e: We applied several filters to ensure question quality:\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e\u003cul\u003e \u003cli\u003e \u003cp\u003eShuffled answer options to prevent bias.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eRemoved questions referring directly to actions (for closed-book testing).\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eApplied Jaccard similarity thresholds to ensure we generated high quality incorrect answers (or discriminators\u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e) – i.e., we removed questions that contained multiple rephrased correct answers (intra-candidate similarity), or where the phrasing of the correct answer reflected that of the question, giving away the answer (extra-candidate similarity).\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eRemoved questions solely based on effectiveness ratings from the CE \u003cem class=\"Highlight ht4fc55b9d-f515-4fa8-9e5d-6731d62f45ba\" highlight=\"true\" htmatch=\"database\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003edatabase\u003c/em\u003e (e.g., \"How effective is ..\").\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e\u003cp\u003eWe deviated from an automated evaluation method\u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e by not applying Item Response \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"theory\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eTheory\u003c/em\u003e for iterative question filtering due to using all post-filtering questions across evaluated LLMs. This gave a large dataset of 1867 unfiltered questions, which whilst potentially containing inaccuracies, provided a basis for comparing relative performance across different LLMs, retrieval strategies, and conservation topic areas (Box 1).\u003c/p\u003e\u003cp\u003e\u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eHuman\u003c/em\u003e Evaluation\u003c/p\u003e\u003cp\u003eWe also curated a filtered dataset of 45 questions (checked for clarity and accuracy – see below) to measure the absolute performance of different LLMs and retrieval strategies and provide a \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehuman\u003c/em\u003e evaluation benchmark. To produce this filtered dataset, we:\u003c/p\u003e\u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eRandomly sampled three actions from each synopsis.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eApplied automated filtering as described above.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eIterated the process to ensure each synopsis had at least one question.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eManually filtered the questions down to a final subset of 45 questions by two reviewers, who were researchers from the Conservation Evidence project team and did not take part in answering the \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"survey\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003esurvey\u003c/em\u003e questions. The filtering was based on three main criteria: 1. the clarity of the question, 2. the accuracy of the answers, and 3. that there was one clear correct answer. Both reviewers independently selected questions to reject and then met to discuss any disagreements. The final subset of 45 questions were then reviewed again by both researchers to check they met all three criteria.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e\u003cp\u003eTo establish a \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehuman\u003c/em\u003e evaluation benchmark, we formed a \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehuman\u003c/em\u003e expert comparison group that represented the entire Conservation Evidence project team (six experts) based at the University of Cambridge who were uniquely qualified to answer questions and retrieve information on the Conservation Evidence \u003cem class=\"Highlight ht4fc55b9d-f515-4fa8-9e5d-6731d62f45ba\" highlight=\"true\" htmatch=\"database\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003edatabase\u003c/em\u003e. Their experience in using the online \u003cem class=\"Highlight ht4fc55b9d-f515-4fa8-9e5d-6731d62f45ba\" highlight=\"true\" htmatch=\"database\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003edatabase\u003c/em\u003e (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e\u003ca href=\"http://www.conservationevidence.com\" target=\"_blank\"\u003ewww.conservationevidence.com\u003c/a\u003e\u003c/span\u003e\u003cspan address=\"http://www.conservationevidence.com\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e) to retrieve information to answer practitioner questions on conservation interventions provided the ideal \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehuman\u003c/em\u003e comparison for this evaluation – i.e., the benchmark they would provide would represent highly expert use and querying of the Conservation Evidence \u003cem class=\"Highlight ht4fc55b9d-f515-4fa8-9e5d-6731d62f45ba\" highlight=\"true\" htmatch=\"database\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003edatabase\u003c/em\u003e to compare LLMs and retrieval strategies against, which could not be achieved by sourcing experts from a more general group of conservation experts.\u003c/p\u003e\u003cp\u003eEach \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehuman\u003c/em\u003e expert answered all 45 multiple choice questions using an online anonymous \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"survey\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003esurvey\u003c/em\u003e implemented via Qualtrics \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"survey\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003esurvey\u003c/em\u003e software. The questions recorded the responses of these participants to these questions and asked them to provide a link to the action webpage (source document) that they used to answer the question (i.e., where they retrieved the evidence from). We also timed how long it took each participant to answer each question using a built-in timing function in Qualtrics \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"survey\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003esurvey\u003c/em\u003e software. We use median time and interquartile range to be conservative as there were some extreme values, probably caused by participants taking breaks during questions.\u003c/p\u003e\u003cp\u003eWe obtained \u003cem class=\"Highlight htf340ff0d-a602-4893-ae8d-b60ea075e112\" highlight=\"true\" htmatch=\"ethic*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eethical\u003c/em\u003e \u003cem class=\"Highlight htf340ff0d-a602-4893-ae8d-b60ea075e112\" highlight=\"true\" htmatch=\"approval\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eapproval\u003c/em\u003e from the University of Cambridge Computing Science \u003cem class=\"Highlight htf340ff0d-a602-4893-ae8d-b60ea075e112\" highlight=\"true\" htmatch=\"ethics committee*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eEthics Committee\u003c/em\u003e (review no. #2324). Please see the Participant Information Sheet and \u003cem class=\"Highlight htf340ff0d-a602-4893-ae8d-b60ea075e112\" highlight=\"true\" htmatch=\"consent*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eConsent\u003c/em\u003e Forms in the Supporting Information.\u003c/p\u003e\u003cp\u003eRetrieval Strategies\u003c/p\u003e\u003cp\u003eWe evaluated three retrieval strategies (Box 1) that are commonly used in RAG systems to retrieve relevant information:\u003c/p\u003e\u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eDense Retrieval\u003c/b\u003e: We used Nomic Embed Text v1.5 through the SentenceTransformers (SBERT) module for embedding. This method represents documents and queries as dense vectors in a high-dimensional space, allowing for semantic similarity comparisons.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eSparse Retrieval\u003c/b\u003e: We employed BM25, a probabilistic retrieval function that ranks documents based on the appearance of query terms, considering term frequency and document length.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eHybrid Retrieval\u003c/b\u003e: We used a cross-encoder (ms-marco-MiniLM-L-6-v2) for re-ranking. This approach combines the strengths of both dense and sparse retrieval, using an initial retrieval step followed by a more computationally intensive re-ranking step.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e\u003cp\u003eAll retrieval strategies selected two action source documents that were added to the prompt sent to the \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e (Table S1).\u003c/p\u003e\u003cp\u003e\u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e Evaluation\u003c/p\u003e\u003cp\u003eWe evaluated the following state-of-the-art LLMs at the time of conducting the study:\u003c/p\u003e\u003cul\u003e \u003cli\u003e \u003cp\u003eLlama 3.1 8B Instruct-Turbo (FP8)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eLlama 3.1 70B Instruct-Turbo (FP8)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eGemma2 Instruct − 9B (BF16)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eGemma2 Instruct − 27B (BF16)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eMixtral 8x22B Instruct (BF16)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eGemini 1.5 Flash (gemini-1.5-flash-001)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eGemini 1.5 Pro (gemini-1.5-pro-001)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eClaude 3.5 Sonnet (claude-3-5-sonnet-20240620)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e\u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"gpt\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eGPT\u003c/em\u003e-4o (\u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"gpt\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003egpt\u003c/em\u003e-4o-2024-08-06)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e\u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"gpt\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eGPT\u003c/em\u003e-4o mini (\u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"gpt\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003egpt\u003c/em\u003e-4o-mini-2024-07-18).\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e\u003cp\u003eWe used Google Cloud’s Vertex AI for Gemini 1.5 Pro and Gemini 1.5 Flash, OpenAI for \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"gpt\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eGPT\u003c/em\u003e-4o and GPT4o mini, Claude for Claude 3.5 Sonnet and Deepinfra for all other models. To maximise the repeatability of \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e responses, we set the temperature hyperparameter to the minimum value of 0 – this effectively minimises the randomness of \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e responses and maximises predictability as much as possible (i.e., almost deterministic). We tested these LLMs under six different exam conditions reflecting different retrieval strategies (or lack of them):\u003c/p\u003e\u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eClosed book (no source documents provided)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eOracle (using the singular source document used to generate the question)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eConfused (oracle document plus one random document from an action with evidence in the same synopsis)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eDense retrieval\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eSparse retrieval\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eHybrid retrieval (top 5 dense + top 5 sparse + reranking to top 2).\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e\u003cp\u003eSee Table S1 for the prompt structure used and examples of prompts.\u003c/p\u003e\u003cp\u003eStatistical testing\u003c/p\u003e\u003cp\u003eFirst, we tested the null hypotheses that there was no difference in the accuracy of each \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e (using a hybrid retrieval strategy) and a random guesser (with a 25% chance) at correctly answering the 45 multiple-choice questions. We used a modified permutation test, with a similar approach to a sign test, whereby we directly compared the correct and incorrect answers given by each \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e versus a random guesser. For questions where the random guesser gave a correct answer and the \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e gave an incorrect answer, a value of -1 was assigned. For the converse, a value of + 1 was assigned (i.e. positive values = \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e wins, negative values = random guesser wins). For \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"case*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ecases\u003c/em\u003e where both the \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e and random guesser got the questions right or wrong (i.e., a draw), a value of 0 was assigned. As there was a 25% chance of there being a draw for each question, we subtracted the number of draws that might have occurred by chance across the 45 questions – this was done by randomly sampling from the numbers 0 and 1 (0 = no draw, 1 = draw, with a probability of 0.75 of no draw and 0.25 of a draw) with replacement 45 times and taking the sum. We calculated a test statistic by taking the sum of the − 1s, +1s, and remaining 0s across the 45 questions as in a conventional sign test. This process was repeated 10,000 times to produce a raw p-value by calculating the proportion of test statistics equal to zero. Since we tested ten null hypotheses (one for each \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e) separately, we used the Holm adjustment to correct p-values for multiplicity. We also calculated the mean, standard deviation, and 95% Confidence Intervals of the permutation test statistic.\u003c/p\u003e\u003cp\u003eSecond, we repeated the permutation test described previously to test the null hypotheses that there was no difference in the accuracy of each \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e and a randomly selected \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehuman\u003c/em\u003e expert at answering multiple-choice questions. We used paired data for both six \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehuman\u003c/em\u003e experts and each \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e (using a hybrid retrieval strategy) for all the 45 filtered questions. For each question, we randomly selected one of the six \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehuman\u003c/em\u003e expert’s answers, comparing its correctness to the correctness of the given \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e’s answer. This process was repeated 1,000,000 times given the large number (6\u003csup\u003e45\u003c/sup\u003e) of possible combinations.\u003c/p\u003e\u003cp\u003eThird, we repeated the permutation test for the null hypotheses that there was no difference in the retrieval accuracy of each of the three retrieval strategies (sparse, dense, and hybrid) and a randomly selected \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehuman\u003c/em\u003e expert. As the number of possible retrieval answers was large (2250 potential action pages to retrieve), we did not need to adjust for the probability of draws by chance as we did previously for the multiple-choice answers. For each question, we randomly selected one of the six \u003cem class=\"Highlight ht71194251-f7a6-4c2d-a145-3d9f25b46662\" highlight=\"true\" htmatch=\"human*\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003ehuman\u003c/em\u003e expert’s retrieval responses, comparing its correctness to the correctness of the given retrieval strategy’s response. This process was repeated 1,000,000 times given the large number (6\u003csup\u003e45\u003c/sup\u003e) of possible combinations.\u003c/p\u003e\u003cp\u003eFinally, we used two separate logistic regression Generalised Linear Models (GLMs) to test the null hypotheses that there were no differences in 1.) the accuracy of different LLMs at answering questions; and 2.) the retrieval accuracy of sparse, dense, and hybrid retrieval strategies. GLM selection was carried out via likelihood ratio tests of nested GLMs using the ‘anova’ function and Aikake Information Criterion corrected for small \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"sample\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003esample\u003c/em\u003e sizes (threshold of \u0026gt; 2 ΔAICc). Our initial GLM consisted of two interaction terms: \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e accuracy ~ Synopsis*\u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e + \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e*Exam Type to test whether certain LLMs performed better for certain synopses, and whether certain LLMs performed better for certain types of exams (e.g., closed book, oracle, hybrid retrieval, etc.). The best GLM selected contained all the explanatory variables but without any interactions: \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e accuracy ~ Synopsis + \u003cem class=\"Highlight ht29216696-c42e-4f00-932a-aea34347df6a\" highlight=\"true\" htmatch=\"llm\" htloopnumber=\"791596286\" style=\"font-style: inherit;\"\u003eLLM\u003c/em\u003e + Exam Type. GLMs for each possible combination of variables were also tested, including a null intercept-only GLM. For the second GLM, we initially started with an interaction term: Retrieval accuracy ~ Synopsis*Retrieval Strategy, but the best GLM selected contained both variables but without an interaction (Retrieval accuracy ~ Synopsis + Retrieval Strategy). GLMs for each possible combination of variables were also tested, including a null intercept-only GLM.\u003c/p\u003e\u003cp\u003eStatistical significance of covariates in the best GLMs were assessed with an analysis of deviance test (Type II) using the ‘Anova’ function from the R package car (Fox and Weisberg 2019). We conducted pairwise tests between the levels of the categorical explanatory variables using the emmeans package with a Tukey adjustment for multiplicity (Lenth 2021).\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eComparison with human experts\u003c/p\u003e \u003cp\u003eOpen-book performance of LLMs demonstrates competitiveness with human expert performance\u003c/p\u003e \u003cp\u003eAcross all 45 selected questions, three LLMs, GPT-4o (97.8%), Llama 3.1 70B (97.8%), and Gemma 2 27B (95.6%) outperformed the average performance of human experts (mean\u0026thinsp;=\u0026thinsp;94.8%; median\u0026thinsp;=\u0026thinsp;95.6%; IQR\u0026thinsp;=\u0026thinsp;93.8\u0026ndash;95.6%; Range\u0026thinsp;=\u0026thinsp;91.1%-97.8%) when provided access to the Conservation Evidence corpus and using a hybrid retrieval strategy (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). All LLMs performed significantly better than a random guesser (means of 27.8\u0026ndash;32.8 more correct answers versus a random guesser; p\u0026thinsp;=\u0026thinsp;0 for all LLMs; Table S2), reflecting the fact that the LLM percentage accuracy was substantially higher than the 25% expected by chance.\u003c/p\u003e \u003cp\u003eMost LLMs had a comparable level of performance at correctly answering questions to the average human (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Mean test statistic values were typically small with narrow 95% Confidence Intervals (CIs) that overlapped with zero, indicative of the large number of draws observed in our datasets (beyond those expected by chance) and thus the highly similar performance of each LLM and a randomly selected human across the 45 questions (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Therefore, for most LLMs, we could not reject the null hypothesis (p\u0026thinsp;\u0026gt;\u0026thinsp;0.05; Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) that there was no difference in the head-to-head performance of the LLM and a randomly selected human expert, except for Llama 3.1 8B Instruct Turbo which performed significantly worse than the human experts (mean\u0026thinsp;=\u0026thinsp;3.7 fewer correct answers; 95% CIs\u0026thinsp;=\u0026thinsp;2\u0026ndash;5 fewer correct answers; p\u0026thinsp;=\u0026thinsp;0.008; Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Gemma2\u0026ndash;9B Instruct also appeared to perform worse than a random human expert (mean\u0026thinsp;=\u0026thinsp;2.7 fewer correct answers; 95% CIs\u0026thinsp;=\u0026thinsp;1\u0026ndash;4 fewer correct answers; Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e), but we could not reject the null hypothesis at the 0.05 significance level due to adjusting p-values for multiple comparisons. Conversely, GPT-4o and Llama 3.1 70B Instruct Turbo almost always matched or exceeded human experts (mean\u0026thinsp;=\u0026thinsp;1.3 more correct answers; 95% CIs\u0026thinsp;=\u0026thinsp;0\u0026ndash;3 more correct answers; Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u0026ndash; Results of a paired comparison of question accuracy of Large Language Models using a hybrid retrieval strategy versus human experts, as well as comparing retrieval accuracy of different retrieval strategies versus human experts for the filtered 45-question set. Test statistics are from a permutation test used to test two null hypotheses: 1. LLM: no difference in multiple-choice question accuracy of the given LLM and a randomly selected human expert; 2. Retrieval strategies: no difference in retrieval accuracy of a retrieval strategy and a randomly selected human expert. The test statistic can be interpreted as follows: negative values\u0026thinsp;=\u0026thinsp;random guesser or randomly selected human expert answered more questions correctly; positive values\u0026thinsp;=\u0026thinsp;LLM/retrieval strategy answered more questions correctly, and zero\u0026thinsp;=\u0026thinsp;all draws or equal numbers of wins and losses. The test statistic accounts for the number of draws expected by chance for the multiple-choice questions. *denotes statistically significant Holm-adjusted p-value at 0.05 significance level.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLarge Language Model\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOverall percentage question accuracy\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHolm-\u003c/p\u003e \u003cp\u003eadjusted p-value\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMean permutation test statistic\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSD permutation test statistic\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eLower 95% CI\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eUpper 95% CI\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGPT-4o\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e97.80%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1.331\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e3.000\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLlama 3.1 70B Instruct Turbo\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e97.80%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1.335\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e3.000\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGemma2\u0026ndash;27B Instruct\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e95.60%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.333\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.999\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e2.000\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGPT-4o Mini\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e93.30%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-0.667\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-2.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMixtral 8x22B\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e93.30%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-0.667\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-2.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eClaude 3.5 Sonnet\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e93.30%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-0.665\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.001\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-2.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGemini 1.5 Flash\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e91.10%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.801\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-1.667\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-3.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.000\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGemini 1.5 Pro\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e91.10%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.801\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-1.668\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-3.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.000\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGemma2\u0026ndash;9B Instruct\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e88.90%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.135\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-2.666\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-4.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-1.000\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLlama 3.1 8B Instruct Turbo\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e86.70%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.008*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-3.668\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-5.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-2.000\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eRetrieval strategy\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003eOverall percentage retrieval accuracy\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003eHolm-\u003c/b\u003e\u003c/p\u003e \u003cp\u003e\u003cb\u003eadjusted p-value\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003eMean permutation test statistic\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003eSD permutation test statistic\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003eLower 95% CI\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003eUpper 95% CI\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDense\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e80.0%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.029*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-3.500\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.404\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-6.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-1.000\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSparse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e71.1%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.000*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-7.501\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.407\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-10.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-5.000\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHybrid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e88.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.270\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.500\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.404\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-2.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e3.000\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eComparison of retrieval strategy performance with expert retrieval\u003c/p\u003e \u003cp\u003eThe hybrid strategy\u0026rsquo;s overall retrieval accuracy (i.e., identifying the correct source document to answer the question) was 88.9%, narrowly outperforming the average of human experts (mean\u0026thinsp;=\u0026thinsp;87.8%; median\u0026thinsp;=\u0026thinsp;88.9%; IQR\u0026thinsp;=\u0026thinsp;85.6\u0026ndash;88.9%; Range\u0026thinsp;=\u0026thinsp;82.2\u0026ndash;93.3%).\u003c/p\u003e \u003cp\u003eUsing equivalent permutation tests as before, we found that the per-question retrieval accuracy of dense and sparse retrieval strategies was significantly worse than a randomly selected human expert (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e \u0026ndash; Dense: mean\u0026thinsp;=\u0026thinsp;3.5 fewer correct retrievals, 95% CIs\u0026thinsp;=\u0026thinsp;1\u0026ndash;6 fewer correct retrievals, p\u0026thinsp;=\u0026thinsp;0.029; Sparse: mean\u0026thinsp;=\u0026thinsp;7.5 fewer correct retrievals, 95% CIs\u0026thinsp;=\u0026thinsp;5\u0026ndash;10 fewer correct retrievals, p\u0026thinsp;\u0026lt;\u0026thinsp;0.001). However, we could not reject the null hypothesis (of no difference in retrieval accuracy versus a random expert) for the hybrid retrieval strategy for which we found comparable levels of accuracy with humans (mean\u0026thinsp;=\u0026thinsp;0.5 more correct retrievals, 95% CIs\u0026thinsp;=\u0026thinsp;2 fewer to 3 more correct retrievals; p\u0026thinsp;=\u0026thinsp;0.271; Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). This was again associated with large numbers of draws between human experts and the hybrid retrieval strategy.\u003c/p\u003e \u003cp\u003eThe median time taken to answer each question by a human expert was 139.5 seconds (IQR 90.9-269.1 seconds) versus practically instantaneous responses from any LLM (given retrieval had already been completed and documents effectively cached).\u003c/p\u003e \u003cp\u003eModel comparisons across unfiltered dataset\u003c/p\u003e \u003cp\u003eQuestion accuracy across synopses, exam conditions, and retrieval strategies\u003c/p\u003e \u003cp\u003eWe found that for the larger dataset of questions, LLM accuracy differed significantly between synopses, LLMs, and exam conditions (Table S3, S4). The accuracy of LLMs was significantly higher for the Biodiversity of Marine Artificial Structures and Natural Pest Control synopses than almost all other synopses \u0026ndash; differences were non-significant with each other and the Bat, Bee, and Natural Pest Control synopses (Table S5, Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Synopses that LLMs performed worst for were Reptile, Control of Freshwater Invasives, and Butterfly and Moth Conservation (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e; Table S4, S5).\u003c/p\u003e \u003cp\u003eGenerally, LLMs had significantly higher accuracy under oracle conditions than all other exam types, followed by confused, hybrid retrieval, dense retrieval, sparse retrieval, and closed book in descending order \u0026ndash; all differences were statistically significant (Table S6).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eGPT-4o had significantly higher accuracy than all other LLMs - although the difference was non-significant for Llama 3.1 70B Instruct Turbo (OR\u0026thinsp;=\u0026thinsp;1.118, z\u0026thinsp;=\u0026thinsp;2.807, p\u0026thinsp;=\u0026thinsp;0.134; Table S7). Llama 3.1 8B had significantly lower accuracy than all LLMs, whilst next worst was Mixtral 8x22 which had significantly lower accuracy than the rest of the LLMs (Table S7).\u003c/p\u003e \u003cp\u003eLLMs exhibited lower performance in closed-book conditions compared to open-book scenarios (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e; Figure S3; Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). When presented with only the correct document all LLMs showed high levels of performance (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e) with some, such as GPT-4o and Llama 3.1-70B reaching 100% answer correctness across the human filtered questions. Performance declined across all LLMs when extraneous, irrelevant documents were introduced in the 'confused' scenario (Figure S3; Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u0026ndash; Overall Large Language Model (LLM) accuracy across different exam conditions on the unfiltered dataset. The table is sorted by LLM performance under the hybrid retrieval strategy. Results for the filtered 45-question dataset are presented in Table S8. We also specify whether LLMs are open or closed source.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"8\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLLM\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eClosed Book\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eOracle\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eConfused\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eDense\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eSparse\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eHybrid\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eOpen/closed source\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGPT-4o\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e69.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e95.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e94.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e86.1%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e83.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e90.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eClosed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLlama 3.1 70B Instruct Turbo\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e68.7%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e94.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e92.7%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e85.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e82.5%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e87.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eOpen\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGemini 1.5 Pro\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e65.5%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e94.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e92.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e84.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e81.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e87.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eClosed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGemma2\u0026ndash;27B Instruct\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e66.1%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e93.7%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e92.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e83.6%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e80.7%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e87.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eOpen\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGPT-4o Mini\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e66.5%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e92.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e91.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e83.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e81.1%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e86.5%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eClosed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGemma2\u0026ndash;9B Instruct\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e63.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e92.5%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e91.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e81.7%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e79.0%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e85.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eOpen\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGemini 1.5 Flash\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e64.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e93.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e91.7%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e82.5%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e80.0%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e85.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eClosed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eClaude 3.5 Sonnet\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e65.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e93.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e91.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e80.7%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e78.1%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e83.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eClosed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMixtral 8x22B\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e64.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e91.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e90.1%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e79.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e75.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e83.0%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eOpen\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLlama 3.1 8B Instruct Turbo\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e62.6%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e90.4%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e86.1%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e77.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e73.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e80.1%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eOpen\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eRetrieval accuracy across synopses, exam conditions, and retrieval strategies\u003c/p\u003e \u003cp\u003eRetrieval accuracy significantly differed across both synopses and retrieval strategies (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e; Table S9, S10). Across all LLMs, hybrid retrieval accuracy was significantly better than both Dense and Sparse retrieval (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e; Table S10, S11), as overall there was an increase in retrieving the correct document from 61.8% and 75.2\u0026ndash;83.2% from Sparse and Dense to Hybrid, respectively. This aligns with the increased accuracy of LLMs with hybrid retrieval strategy compared to LLMs with dense or sparse retrieval strategies (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e; Fig. S3; Table S6).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u0026ndash; Overall retrieval accuracy of different retrieval strategies across the unfiltered dataset of questions.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRetrieval strategy\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePercentage of questions where source action document retrieved correctly\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePercentage of questions where document from same synopsis was retrieved (considers both selected documents)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePercentage of questions where source action document was selected first (if one of two documents selected was correct)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDense\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e75.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e85.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e80.0%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSparse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e61.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e62.7%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e82.6%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHybrid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e83.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e84.7%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e78.2%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eAcross all synopses, hybrid retrieval accuracy was significantly greater than both dense (OR\u0026thinsp;=\u0026thinsp;1.66, z\u0026thinsp;=\u0026thinsp;6.17, p\u0026thinsp;\u0026lt;\u0026thinsp;0.0001; Table S11) and sparse retrieval (OR\u0026thinsp;=\u0026thinsp;3.14, z\u0026thinsp;=\u0026thinsp;14.5, p\u0026thinsp;\u0026lt;\u0026thinsp;0.0001; Table S11), as was found for LLM accuracy using dense versus sparse retrieval (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e; Table S6). Dense retrieval accuracy was also significantly greater than sparse retrieval (OR\u0026thinsp;=\u0026thinsp;1.89, z\u0026thinsp;=\u0026thinsp;8.75, p\u0026thinsp;\u0026lt;\u0026thinsp;0.0001; Table S11). Dense retrieval resulted in the correct (as used to generate the question) document for 75.2% of questions whilst this was only 61.8% for sparse retrieval (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). Notably, in 85.8% of questions, dense retrieval returned at least one document from the same synopsis of the source document, for Sparse this was 62.7% - only marginally higher than returning the correct document.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eFor synopses, the poorest retrieval accuracy was found for the Mediterranean Farmland synopsis and significantly lower than more than half of the synopses (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e; Table S9). Several synopses (e.g., Marine and Freshwater Mammals, Marine Artificial Structures, Subtidal Benthic Invertebrates) had significantly higher retrieval accuracy than some of the synopses with the lowest retrieval accuracies (e.g., Reptile Conservation and Mediterranean Farmland; Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e; Table S12).\u003c/p\u003e \u003cp\u003eGenerational Progress\u003c/p\u003e \u003cp\u003eTo assess the improvement of cutting-edge LLMs over time, we tested GPT-3.5 Turbo Instruct (released in September 2023 but derived from GPT-3.5 originally released in November 2022) on 679 unfiltered questions (Table S13). Performance in the Oracle scenario was notably low for GPT-3.5 Turbo Instruct compared to all ten, more recent LLMs (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e; Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eOur findings that most LLMs tested using a hybrid retrieval strategy, in particular GPT-4o and Llama 3.1 70B Instruct Turbo, demonstrated comparable performance at answering multiple choice questions with a randomly selected human expert and far exceeded performance expected from random guessing. In addition, retrieval accuracy for the hybrid strategy was also competitive with human expert retrieval. The retrieval performance was found to be lower than answer correctness for both LLMs and humans, likely due to overlapping key messages between actions, which enables related actions to inform responses to other questions. An example of this phenomenon is provided in Figure S1.\u003c/p\u003e \u003cp\u003eOur findings also suggest that the closed-book performance of LLMs (without any retrieval strategy) demonstrated a level of conservation-specific knowledge, as evidenced by the fact that all LLMs performed better than random guessing under all exam conditions. This varied by topic area (Conservation Evidence synopses) but results remained high enough to suggest that the pre-training corpora incorporated relevant conservation literature \u0026ndash; whilst many studies are published behind paywalls, the Conservation Evidence database website (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.conservationevidence.com\u003c/span\u003e\u003cspan address=\"https://www.conservationevidence.com\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e)\u003c/span\u003e and synopses are freely available online.\u003c/p\u003e \u003cp\u003eWe also found that the performance of LLMs was consistent between the following exam conditions: oracle (providing the correct source action document) and confused (providing both a correct source action document with a random one). Since retrieval always results in two documents being provided to the LLM, the confused exam condition sets a ceiling for their performance \u0026ndash; as expected, all LLMs have lower open-book performance (using hybrid, dense, or sparse retrieval) than in oracle or confused performance across the larger unfiltered dataset. The hybrid retrieval strategy was found to perform substantially better than both dense and sparse retrieval strategies in terms of both question and retrieval accuracy, with dense also outperforming sparse. We also found that whilst the dense retrieval strategy\u0026rsquo;s performance was higher when only considering whether it returned at least one document from the same synopsis of the source document, there was negligible improvement for the sparse retrieval strategy. This is likely attributable to actions within the same synopsis being semantically similar and dense embeddings being able to capture this.\u003c/p\u003e \u003cp\u003eFinally, our findings also suggest there has been substantial generational improvement in LLM performance over a period of approximately two years, given that GPT-3.5 Turbo Instruct (derived from GPT-3.5 released in 2022) performed notably poorer than all ten more recently developed LLMs that we tested. This includes against models such as GPT-4o mini and Gemini 1.5 Flash which are considerably smaller and cheaper. Therefore, there does appear to be the potential for further improvements to the models we tested over the next few years, and emphasises the importance of our validation pipeline approach to rapidly test new models as they are released and calibrate them against expert-level training data.\u003c/p\u003e \u003cp\u003eLimitations and future work\u003c/p\u003e \u003cp\u003eThese findings need to be considered in the context of certain limitations. First, our results relate to LLMs\u0026rsquo; specific performance on the Conservation Evidence database, which may not generalise to questions on other conservation questions. We also may not have captured the full range of questions that decision-makers have as the questions were derived from the Conservation Evidence database and generated by a LLM - thus any gaps and biases in the CE database and the LLM\u0026rsquo;s pre-training data, including geographic, taxonomic, and language bias \u003csup\u003e\u003cspan additionalcitationids=\"CR32\" citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e\u003c/sup\u003e may influence the questions set and thus the observed performance of LLMs. Nevertheless, we found that the observed performance of LLMs was broadly similar between the human-filtered questions and the larger dataset of unfiltered questions. However, LLMs are ultimately black box models, and we have a poor understanding of the pre-training data used to build them and their associated biases \u0026ndash; thus further testing across different subject areas and domains remains important. Our results suggest that performance for certain synopses was poorer than others, although this did not necessarily follow an obvious taxonomic or biogeographical pattern. Our approach could be used to test LLM performance when retrieving information from databases, repositories and syntheses that already exist in other disciplines and topic areas \u0026ndash; e.g., web databases such as the IUCN Red List and Education Endowment Foundation Toolkits that could be treated in a similar way to how the Conservation Evidence database was in our study. It is also important to note that our evaluation method used Claude 3.5 Sonnet to generate the questions and answers and so all our questions should have been framed in a manner answerable by LLMs. However, Claude 3.5 Sonnet only performed in the middle to lower end of other LLMs tested, suggesting that it received no significant advantage in answering questions over the others. We must also acknowledge that despite setting the hyperparameter \u0026lsquo;temperature\u0026rsquo; of all LLMs to zero, we cannot completely guarantee that LLM responses were 100% deterministic. However, whilst it is possible that if prompted a large number of times, LLMs may have provided a different response to a small number of questions, this is unlikely to have altered our major findings.\u003c/p\u003e \u003cp\u003eThe human expert group we used was also limited in its sample size because we focused on testing LLMs against humans that were highly familiar with navigating and answering questions using the Conservation Evidence database. However, this means that the human expert group we used is likely to represent an extremely high threshold of human performance compared to that of a more diverse group of experts with conservation expertise but with less expertise on using the database or synthesising evidence on conservation interventions. In this context, however, the LLM performance we observed is even more impressive. We may have also underestimated the retrieval times for human experts given that the questions we generated were based on a single conservation action and thus had a relatively high level of specificity \u0026ndash; it may have taken experts longer to answer questions that rely on synthesising information across multiple conservation actions, and certainly would have taken a wider group of conservation experts without specific knowledge of the database more time too. Therefore, there are considerable potential time savings to be achieved by querying the Conservation Evidence database via a LLM-based RAG system over asking a human expert.\u003c/p\u003e \u003cp\u003eFinally, our exam-style evaluation scheme was designed to provide a plausible upper bound on LLMs performance at answering questions as a first step to evaluating their suitability for decision support when drawing upon evidence databases and syntheses. There is likely to be more potential for errors when answering questions using free text generation as opposed to the constrained multiple-choice answers we examined. Furthermore, our evaluation only included a single task: answering a multiple-choice question on a single conservation action from a single synopsis. Now that we have demonstrated LLMs using a hybrid retrieval strategy are competitive with human experts at simple retrieval and question answering tasks, future research should investigate at which point LLM performance declines as tasks become more complex in terms of both questions and responses (e.g., questions requiring more synthesis across multiple actions or synopses with nuanced answers). Once this performance threshold with task complexity is identified, future systems should then be designed to stay above that threshold to avoid the risk of misinforming decision-makers. However, there is also the potential to test whether more advanced prompt engineering can improve the performance of LLMs, as well as more computationally expensive embedding approaches that might improve retrieval performance.\u003c/p\u003e \u003cp\u003eRisks\u003c/p\u003e \u003cp\u003eIt is also important to consider the ethics and clear associated risks of using LLMs in a decision support capacity to answer conservation questions based on evidence databases. First, equitable access to using LLMs for these purposes is important \u0026ndash; any implementation should ideally be free and open access. This may also place an important constraint on which LLMs can be used in terms of cost \u0026ndash; therefore, it is important to consider the relative cost-effectiveness of LLMs by comparing their performance to their cost. Indeed, we found that many open-source LLMs performed just as well as closed source LLMs (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). Furthermore, access to LLMs will always be constrained, at least in part, by access to technology, power, and stable internet connections \u0026ndash; although most decision support tools should be able to be run on relatively cheap devices, such as mobile phones. Therefore, the consideration of the power dynamics between Global North and Global South institutions and organisations owning and using these tools should be carefully considered and recognised to avoid unequitable relationships at the research-practice and research-policy interfaces\u003csup\u003e\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eIt is also important to recognise that the training of LLMs comes with substantial embodied environmental costs. These environmental costs have already occurred prior to the release of these models (i.e., in their initial training) and so there is arguably an ethical case that once developed LLMs should be used as much as possible for tasks that might mitigate these environmental costs. However, there are still costs associated with LLM inference via APIs (although these are a fraction of the original training costs) and so efficiency in the design of LLM-based RAG systems needs to be prioritised \u0026ndash; thus we should make sure that if there is a choice between using models with similar levels of performance, that we select the simpler, computationally cheaper models with lower environmental costs whenever possible.\u003c/p\u003e \u003cp\u003eThere is also a risk that LLMs may lead to deskilling of conservation practitioners and scientists in searching for, and assessing, the evidence. Although the goal is to make accessing and interpreting evidence databases and syntheses easier and more efficient, there is the risk that practitioners and scientists may think less critically about the evidence, its limitations, source, and validity. This could also lead to a lack of accountability in the use of evidence, including biased evidence use to support self-serving lines of argument and political rhetoric (e.g., by organisations for greenwashing). Therefore, it will be important to carefully design AI-assisted decision support systems to ensure they prompt users to understand the uncertainty and limitations associated with the evidence base, including its reliability and local relevance and transferability\u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e,\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e,\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e,\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e,\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eWe have shown that LLMs have comparable performance as human experts at providing restricted responses and retrieving relevant information to conservation intervention questions using the Conservation Evidence database. This first step in evaluating the performance of LLMs at decision support tasks now enables research to move on to establishing where any performance issues may occur when LLMs answer more complex, nuanced questions with open-ended, free-text responses. We stress that if performance issues are observed, we should design systems accordingly above this threshold to ensure the risk of misinforming decision-makers is minimised (with comparable or lower levels of error to human experts). For the Conservation Evidence database, our current findings suggest that it is justifiable to implement an LLM capable of providing an intelligent search function, to help direct users of the website to the most suitable page where evidence exists to answer their question. Our findings currently suggest that general LLMs used \u0026lsquo;out-of-the-box\u0026rsquo; are likely to perform poorly at giving evidence-based advice or recommendations for decision-making. Any decisions based on such information are likely to be misinformed. Therefore, we urge those thinking of using general LLMs \u0026lsquo;as-is\u0026rsquo; to provide decision support within their organisation or to end users for certain tasks, that they spend time and effort to carefully consider how to design and evaluate LLM-based systems using retrieval augmented generation (RAG) and a hybrid retrieval strategy.\u003c/p\u003e \u003cp\u003eLooking to the future, if the rapid pace of improvements to underlying models continue, it is possible that better performing LLMs coupled with further refinements of prompts and retrieval strategies can enable the development of RAG systems capable of expert-level, evidence-based advice based on evidence databases and syntheses. However, careful and considered evaluation of LLM-based decision support systems is needed along the way to ensure that more rapid and intuitive access to relevant evidence to inform practice and policy does not come at the cost of misinforming decision-makers with biased, erroneous, or misleading information.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e \u003ch2\u003eConflict of interest disclosure:\u003c/h2\u003e \u003cp\u003eThe authors have no conflicts of interest to disclose.\u003c/p\u003e \u003c/p\u003e\u003ch2\u003eFunding statement:\u003c/h2\u003e \u003cp\u003eRI was supported by a UROP internship at the University of Cambridge. APC received financial support from Imperial College London through an Imperial College Research Fellowship grant, as well as a Henslow Fellowship funded by the Cambridge Philosophical Society.\u003c/p\u003e\u003ch2\u003eAuthors contributions:\u003c/h2\u003e \u003cp\u003eRI: Formal Analysis, Data Curation, Investigation, Methodology, Software, Validation, Writing \u0026ndash; Original Draft Preparation, Writing \u0026ndash; Review \u0026amp; Editing.\u003c/p\u003e\u003ch2\u003eData availability statement:\u003c/h2\u003e \u003cp\u003eData and code associated with this paper can be found at: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.5281/zenodo.14049626\u003c/span\u003e\u003cspan address=\"10.5281/zenodo.14049626\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eLecl\u0026egrave;re D et al (2020) Bending the curve of terrestrial biodiversity needs an integrated strategy. Nature 585:551\u0026ndash;556\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSutherland WJ, Pullin AS, Dolman PM, Knight TM (2004) The need for evidence-based conservation. Trends Ecol Evol 19:305\u0026ndash;308\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSutherland WJ, Wordley CF (2017) R. Evidence complacency hampers conservation. Nat Ecol Evol 1:1215\u0026ndash;1216\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWalsh JC, Dicks LV, Sutherland WJ (2015) The effect of scientific evidence on conservation practitioners\u0026rsquo; management decisions. Conserv Biol 29:88\u0026ndash;98\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMacLeod CJ, Brandt AJ, Dicks LV (2022) Facilitating the wise use of experts and evidence to inform local environmental decisions. People Nat 4:904\u0026ndash;917\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCook CN, Nichols SJ, Webb JA, Fuller RA, Richards RM (2017) Simplifying the selection of evidence synthesis methods to inform environmental decisions: A guide for decision makers and scientists. Biol Conserv 213:135\u0026ndash;145\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWhite H (2019) The twenty-first century experimenting society: the four waves of the evidence revolution. Palgrave Commun 5:47\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCooke SJ et al (2023) Environmental evidence in action: on the science and practice of evidence synthesis and evidence-based decision-making. Environ Evid 12:10\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKadykalo AN et al (2021) Bridging research and practice in conservation. Conserv Biol 35:1725\u0026ndash;1737\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWalsh JC, Dicks LV, Raymond CM, Sutherland WJ (2019) A typology of barriers and enablers of scientific evidence use in conservation practice. J Environ Manage 250:109481\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRose DC et al (2018) The major barriers to evidence-informed conservation policy and possible solutions. Conserv Lett 11\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHager P et al (2024) Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med 30:2613\u0026ndash;2622\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSandmann S, Riepenhausen S, Plagwitz L, Varghese J (2024) Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks. Nat Commun 15:2050\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eScheepens D, Millard J, Farrell M, Newbold T (2024) Large language models help facilitate the automated synthesis of information on potential pest controllers. Methods Ecol Evol 15:1261\u0026ndash;1273\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSpillias S et al (2024) Human-AI collaboration to identify literature for evidence synthesis. Cell Rep Sustain 1:100132\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBerger-Tal O et al (2024) Leveraging AI to improve evidence synthesis in conservation. Trends Ecol Evol 39:548\u0026ndash;557\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSworna ZT, Urzedo D, Hoskins AJ, Robinson CJ (2024) The ethical implications of Chatbot developments for conservation expertise. AI Ethics. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s43681-024-00460-3\u003c/span\u003e\u003cspan address=\"10.1007/s43681-024-00460-3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKaiser KN et al (2024) Accuracy and consistency of publicly available Large Language Models as clinical decision support tools for the management of colon cancer. J Surg Oncol. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1002/jso.27821\u003c/span\u003e\u003cspan address=\"10.1002/jso.27821\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWeidinger L et al (2022) Taxonomy of Risks posed by Language Models. in. \u003cem\u003eACM Conference on Fairness, Accountability, and Transparency\u003c/em\u003e 214\u0026ndash;229 (ACM, New York, NY, USA, 2022). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1145/3531146.3533088\u003c/span\u003e\u003cspan address=\"10.1145/3531146.3533088\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDe Angelis L et al (2023) ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front Public Health 11\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAu Yeung J et al (2023) AI chatbots not yet ready for clinical use. Front Digit Health 5\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSchwartz IS, Link KE, Daneshjou R (2024) Cort\u0026eacute;s-Penfield, N. Black Box Warning: Large Language Models and the Future of Infectious Diseases Consultation. Clin Infect Dis 78:860\u0026ndash;866\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShen Y et al (2023) ChatGPT and Other Large Language Models Are Double-edged Swords. Radiology 307\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMartin TG et al (2012) Acting fast helps avoid extinction. Conserv Lett 5:274\u0026ndash;280\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLeyer M, Schneider S (2021) Decision augmentation and automation with artificial intelligence: Threat or opportunity for managers? Bus Horiz 64:711\u0026ndash;724\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAdam H, Balagopalan A, Alsentzer E, Christia F, Ghassemi M (2022) Mitigating the impact of biased artificial intelligence in emergency decision-making. Commun Med 2:149\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAugenstein I et al (2023) Factuality Challenges in the Era of Large Language Models\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSalemi A, Kallumadi S, Zamani H (2024) Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation. in \u003cem\u003eProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval\u003c/em\u003e 752\u0026ndash;762ACM, New York, NY, USA. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1145/3626772.3657783\u003c/span\u003e\u003cspan address=\"10.1145/3626772.3657783\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eQuigley K, Koker T, Taylor J, Mancuso V, Brattain L (2024) Designing Retrieval-Augmented Language Models for Clinical Decision Support. 159\u0026ndash;171. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/978-3-031-63592-2_13\u003c/span\u003e\u003cspan address=\"10.1007/978-3-031-63592-2_13\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGuinet G, Omidvar-Tehrani B, Deoras A, Callot L (2024) Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChristie AP et al (2020) Poor availability of context-specific evidence hampers decision-making in conservation. Biol Conserv 248\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChristie AP et al (2020) The challenge of biased evidence in conservation. Conserv Biol. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1111/cobi.13577\u003c/span\u003e\u003cspan address=\"10.1111/cobi.13577\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAmano T et al (2021) Tapping into non-English-language science for the conservation of global biodiversity. PLoS Biol 19:e3001296\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMiller J, White TB, Christie AP (2023) Parachute conservation: Investigating trends in international research. Conserv Lett 16\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChristie AP et al (2019) Simple study designs in ecology produce inaccurate estimates of biodiversity responses. J Appl Ecol 56:2742\u0026ndash;2754\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChristie AP et al (2020) Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences. Nat Commun 11:6377\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"large language models, retrieval augmented generation, retrieval, evidence, evidence-based, conservation, decision support","lastPublishedDoi":"10.21203/rs.3.rs-5409185/v2","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5409185/v2","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eWise use of evidence to support efficient conservation action is key to tackling biodiversity loss with limited time and resources. Evidence syntheses provide key recommendations for conservation decision-makers by assessing and summarising evidence, but are not always easy to access, digest, and use. Recent advances in Large Language Models (LLMs) present both opportunities and risks in enabling faster and more intuitive systems to access evidence syntheses and databases. Such systems for natural language search and open-ended evidence-based responses are pipelines comprising many components. Most critical of these components are the LLM used and how evidence is retrieved from the database. We evaluate the performance of ten LLMs across six different database retrieval strategies against human experts in answering synthetic multiple-choice question exams on the effects of conservation interventions using the Conservation Evidence database. We found that LLM performance was comparable with human experts over 45 filtered questions, both in correctly answering them and retrieving the document used to generate them. Across 1867 unfiltered questions, LLM performance demonstrated a level of conservation-specific knowledge, but this varied across topic areas. A hybrid retrieval strategy that combines keywords and vector embeddings performed best by a substantial margin. We also tested against a state-of-the-art previous generation LLM which was outperformed by all ten current models - including smaller, cheaper models. Our findings suggest that, with careful domain-specific design, LLMs could potentially be powerful tools for enabling expert-level use of evidence syntheses and databases. However, general LLMs used \u0026lsquo;out-of-the-box\u0026rsquo; are likely to perform poorly and misinform decision-makers. By establishing that LLMs exhibit comparable performance with human synthesis experts on providing restricted responses to queries of evidence syntheses and databases, future work can build on our approach to quantify LLM performance in providing open-ended responses.\u003c/p\u003e","manuscriptTitle":"Careful design of Large Language Model pipelines enables expert-level retrieval of evidence-based information from conservation syntheses","msid":"","msnumber":"","nonDraftVersions":[{"code":2,"date":"2025-01-23 15:06:37","doi":"10.21203/rs.3.rs-5409185/v2","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}},{"code":1,"date":"2024-11-13 06:42:59","doi":"10.21203/rs.3.rs-5409185/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"c210ece6-aafc-4b4d-a83b-a930c7e47f77","owner":[],"postedDate":"January 23rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":42782825,"name":"Biological sciences/Ecology/Conservation biology"},{"id":42782826,"name":"Earth and environmental sciences/Ecology/Conservation biology"},{"id":42782827,"name":"Scientific community and society/Scientific community/Policy"},{"id":42782828,"name":"Social science/Science, technology and society"},{"id":42782829,"name":"Scientific community and society/Social sciences/Decision making"}],"tags":[],"updatedAt":"2024-12-16T23:45:17+00:00","versionOfRecord":[],"versionCreatedAt":"2025-01-23 15:06:37","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v2","identity":"rs-5409185","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5409185","identity":"rs-5409185","version":["v2"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}