{"paper_id":"2d58b9cd-5d5b-4b7e-9984-ccf7fd22b407","body_text":"BioLLMBench: A Comprehensive Benchmarking of Large Language Models in Bioinformatics | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Analysis BioLLMBench: A Comprehensive Benchmarking of Large Language Models in Bioinformatics Serghei Mangul, Varuni Sarwal, Viorel Munteanu, Timur Suhodolschi, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3780193/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Large Language Models (LLMs) have shown great promise in their knowledge integration and problem-solving capabilities, but their ability to assist in bioinformatics research has not been systematically evaluated. To bridge this gap, we present BioLLMBench, a novel benchmarking framework coupled with a scoring metric scheme for comprehensively evaluating LLMs in solving bioinformatics tasks. Through BioLLMBench, we conducted a thorough evaluation of 2,160 experimental runs of the three most widely used models, GPT-4, Bard and LLaMA, focusing on 36 distinct tasks within the field of bioinformatics. The tasks come from six key areas of emphasis within bioinformatics that directly relate to the daily challenges and tasks faced by individuals within the field. These areas are domain expertise, mathematical problem-solving, coding proficiency, data visualization, summarizing research papers, and developing machine learning models. The tasks also span across varying levels of complexity, ranging from fundamental concepts to expert-level challenges. Each key area was evaluated using seven specifically designed task metrics, which were then used to conduct an overall evaluation of the LLM’s response. To enhance our understanding of model responses under varying conditions, we implemented a Contextual Response Variability Analysis. Our results reveal a diverse spectrum of model performance, with GPT-4 leading in all tasks except mathematical problem solving. GPT4 was able to achieve an overall proficiency score of 91.3% in domain knowledge tasks, while Bard excelled in mathematical problem-solving with a 97.5% success rate. While GPT-4 outperformed in machine learning model development tasks with an average accuracy of 65.32%, both Bard and LLaMA were unable to generate executable end-to-end code. All models faced considerable challenges in research paper summarization, with none of them exceeding a 40% score in our evaluation using the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) score, highlighting a significant area for future improvement. We observed an increase in model performance variance when using a new chatting window compared to using the same chat, although the average scores between the two contextual environments remained similar. Lastly, we discuss various limitations of these models and acknowledge the risks associated with their potential misuse. Biological sciences/Computational biology and bioinformatics Biological sciences/Computational biology and bioinformatics/Machine learning Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {\"props\":{\"pageProps\":{\"initialData\":{\"identity\":\"rs-3780193\",\"acceptedTermsAndConditions\":true,\"allowDirectSubmit\":true,\"archivedVersions\":[],\"articleType\":\"Analysis\",\"associatedPublications\":[],\"authors\":[{\"id\":267235020,\"identity\":\"eaa4e169-393f-4d8e-8451-f3b775ddc3a6\",\"order_by\":0,\"name\":\"Serghei Mangul\",\"email\":\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABA0lEQVRIiWNgGAWjYJACZgYGCSizwAZIMDYeIEGLQRpISwMxWmDA4DCYwqtFvr338euCCgsG3fazh1/zGJy3W9t+GGhLjU00Li0GZ46bWc84I8FgdiYvzZrH4HbytjOJQC3H0nIbcGmRSGMz5m0DajmQY2YM0mJ2AKiFseEwTi3y858BtfwDajn/BqTlXLLZ+Yf4tTDcYGN+zNsA1HIjx/gxj8EBO7MbBGwxOJPGxsxzTILH7MYbM8Y5BskJZjeAtiTg8Yt8+zHmzzw1dXJm53OMP7ypsLM3O5/+8MGHGhvcDmNgYAPFIw+MkQhWmYBbOQgwf0Bm2ONXPApGwSgYBSMRAACjtV0Qw4TKyQAAAABJRU5ErkJggg==\",\"orcid\":\"https://orcid.org/0000-0003-4770-3443\",\"institution\":\"University of Southern California\",\"correspondingAuthor\":true,\"prefix\":\"\",\"firstName\":\"Serghei\",\"middleName\":\"\",\"lastName\":\"Mangul\",\"suffix\":\"\"},{\"id\":267235021,\"identity\":\"775f962c-6f23-4763-8840-b1555809bc73\",\"order_by\":1,\"name\":\"Varuni Sarwal\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"UCLA\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Varuni\",\"middleName\":\"\",\"lastName\":\"Sarwal\",\"suffix\":\"\"},{\"id\":267235022,\"identity\":\"b8050d7a-348f-4911-b99b-af0fbbf8cbe6\",\"order_by\":2,\"name\":\"Viorel Munteanu\",\"email\":\"\",\"orcid\":\"https://orcid.org/0000-0002-4133-5945\",\"institution\":\"\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Viorel\",\"middleName\":\"\",\"lastName\":\"Munteanu\",\"suffix\":\"\"},{\"id\":267235023,\"identity\":\"d1e7669f-7f24-469d-8466-fa15f2ee078e\",\"order_by\":3,\"name\":\"Timur Suhodolschi\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Technical University of Moldova\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Timur\",\"middleName\":\"\",\"lastName\":\"Suhodolschi\",\"suffix\":\"\"},{\"id\":267235024,\"identity\":\"5d028ace-3923-4024-ac84-2e79f41711ec\",\"order_by\":4,\"name\":\"Dumitru Ciorba\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Technical University of Moldova\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Dumitru\",\"middleName\":\"\",\"lastName\":\"Ciorba\",\"suffix\":\"\"},{\"id\":267235025,\"identity\":\"fdb9171d-f4ab-4c90-8fc3-d7cb04bc69b0\",\"order_by\":5,\"name\":\"Eleazar Eskin\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"University of California, Los Angeles\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Eleazar\",\"middleName\":\"\",\"lastName\":\"Eskin\",\"suffix\":\"\"},{\"id\":267235026,\"identity\":\"3614a98e-8455-4b9c-8c24-537cb878b5df\",\"order_by\":6,\"name\":\"Wei Wang\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"UCLA\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Wei\",\"middleName\":\"\",\"lastName\":\"Wang\",\"suffix\":\"\"}],\"badges\":[],\"createdAt\":\"2023-12-20 06:25:54\",\"currentVersionCode\":1,\"declarations\":\"\",\"doi\":\"10.21203/rs.3.rs-3780193/v1\",\"doiUrl\":\"https://doi.org/10.21203/rs.3.rs-3780193/v1\",\"draftVersion\":[],\"editorialEvents\":[],\"editorialNote\":\"\",\"failedWorkflow\":false,\"files\":[{\"id\":51410989,\"identity\":\"57b9fd51-933e-44e5-b3ff-76859f5b2758\",\"added_by\":\"auto\",\"created_at\":\"2024-02-21 05:08:57\",\"extension\":\"pdf\",\"order_by\":1,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"manuscript-pdf\",\"size\":2951005,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"BioLLMBench.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-3780193/v1_covered_f4e22c7c-ba89-4333-9599-04b0f50cd90a.pdf\"}],\"financialInterests\":\"There is \\u003cb\\u003eNO\\u003c/b\\u003e Competing Interest.\",\"formattedTitle\":\"BioLLMBench: A Comprehensive Benchmarking of Large Language Models in Bioinformatics\",\"fulltext\":[],\"fulltextSource\":\"\",\"fullText\":\"\",\"funders\":[],\"hasAdminPriorityOnWorkflow\":false,\"hasManuscriptDocX\":false,\"hasOptedInToPreprint\":true,\"hasPassedJournalQc\":\"\",\"hasAnyPriority\":true,\"hideJournal\":true,\"highlight\":\"\",\"institution\":\"\",\"isAcceptedByJournal\":false,\"isAuthorSuppliedPdf\":true,\"isDeskRejected\":\"\",\"isHiddenFromSearch\":false,\"isInQc\":false,\"isInWorkflow\":false,\"isPdf\":true,\"isPdfUpToDate\":true,\"isWithdrawnOrRetracted\":false,\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"researchsquare\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":true,\"externalIdentity\":\"\",\"sideBox\":\"\",\"snPcode\":\"\",\"submissionUrl\":\"/submission\",\"title\":\"Research Square\",\"twitterHandle\":\"researchsquare\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"\",\"reportingPortfolio\":\"\",\"inReviewEnabled\":false,\"inReviewRevisionsEnabled\":true},\"keywords\":\"\",\"lastPublishedDoi\":\"10.21203/rs.3.rs-3780193/v1\",\"lastPublishedDoiUrl\":\"https://doi.org/10.21203/rs.3.rs-3780193/v1\",\"license\":{\"name\":\"CC BY 4.0\",\"url\":\"https://creativecommons.org/licenses/by/4.0/\"},\"manuscriptAbstract\":\"Large Language Models (LLMs) have shown great promise in their knowledge integration and problem-solving capabilities, but their ability to assist in bioinformatics research has not been systematically evaluated. To bridge this gap, we present BioLLMBench, a novel benchmarking framework coupled with a scoring metric scheme for comprehensively evaluating LLMs in solving bioinformatics tasks. Through BioLLMBench, we conducted a thorough evaluation of 2,160 experimental runs of the three most widely used models, GPT-4, Bard and LLaMA, focusing on 36 distinct tasks within the field of bioinformatics. The tasks come from six key areas of emphasis within bioinformatics that directly relate to the daily challenges and tasks faced by individuals within the field. These areas are domain expertise, mathematical problem-solving, coding proficiency, data visualization, summarizing research papers, and developing machine learning models. The tasks also span across varying levels of complexity, ranging from fundamental concepts to expert-level challenges. Each key area was evaluated using seven specifically designed task metrics, which were then used to conduct an overall evaluation of the LLM’s response. To enhance our understanding of model responses under varying conditions, we implemented a Contextual Response Variability Analysis. Our results reveal a diverse spectrum of model performance, with GPT-4 leading in all tasks except mathematical problem solving. GPT4 was able to achieve an overall proficiency score of 91.3% in domain knowledge tasks, while Bard excelled in mathematical problem-solving with a 97.5% success rate. While GPT-4 outperformed in machine learning model development tasks with an average accuracy of 65.32%, both Bard and LLaMA were unable to generate executable end-to-end code. All models faced considerable challenges in research paper summarization, with none of them exceeding a 40% score in our evaluation using the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) score, highlighting a significant area for future improvement. We observed an increase in model performance variance when using a new chatting window compared to using the same chat, although the average scores between the two contextual environments remained similar. Lastly, we discuss various limitations of these models and acknowledge the risks associated with their potential misuse.\",\"manuscriptTitle\":\"BioLLMBench: A Comprehensive Benchmarking of Large Language Models in Bioinformatics\",\"msid\":\"\",\"msnumber\":\"\",\"nonDraftVersions\":[{\"code\":1,\"date\":\"2024-01-16 08:13:58\",\"doi\":\"10.21203/rs.3.rs-3780193/v1\",\"editorialEvents\":[{\"type\":\"communityComments\",\"content\":0}],\"status\":\"published\",\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"researchsquare\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":true,\"externalIdentity\":\"\",\"sideBox\":\"\",\"snPcode\":\"\",\"submissionUrl\":\"/submission\",\"title\":\"Research Square\",\"twitterHandle\":\"researchsquare\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"\",\"reportingPortfolio\":\"\",\"inReviewEnabled\":false,\"inReviewRevisionsEnabled\":true}}],\"origin\":\"\",\"ownerIdentity\":\"950aae61-5883-44e7-8ce8-4b6432026e9f\",\"owner\":[],\"postedDate\":\"January 16th, 2024\",\"published\":true,\"recentEditorialEvents\":[],\"rejectedJournal\":[],\"revision\":\"\",\"amendment\":\"\",\"status\":\"posted\",\"subjectAreas\":[{\"id\":28153159,\"name\":\"Biological sciences/Computational biology and bioinformatics\"},{\"id\":28153160,\"name\":\"Biological sciences/Computational biology and bioinformatics/Machine learning\"}],\"tags\":[],\"updatedAt\":\"2024-02-21T05:00:36+00:00\",\"versionOfRecord\":[],\"versionCreatedAt\":\"2024-01-16 08:13:58\",\"video\":\"\",\"vorDoi\":\"\",\"vorDoiUrl\":\"\",\"workflowStages\":[]},\"version\":\"v1\",\"identity\":\"rs-3780193\",\"journalConfig\":\"researchsquare\"},\"__N_SSP\":true},\"page\":\"/article/[identity]/[[...version]]\",\"query\":{\"redirect\":\"/article/rs-3780193\",\"identity\":\"rs-3780193\",\"version\":[\"v1\"]},\"buildId\":\"qtupq5eGEP_6zYnWcrvyt\",\"isFallback\":false,\"isExperimentalCompile\":false,\"dynamicIds\":[84888],\"gssp\":true,\"scriptLoader\":[]}","source_license":"CC-BY-4.0","license_restricted":false}