{"paper_id":"0d611534-e49e-4b68-a7aa-af3ae3aa41db","body_text":"Open-Rosalind: Tool-First Biomedical LLM Agents\nwith Process-Aware Benchmarking\nLiang Wang1,*\n1Huazhong University of Science and Technology; wangliang.f@gmail.com\n*Correspondence: wangliang.f@gmail.com; School of Artificial Intelligence and Automation,\nHuazhong University of Science and Technology, 430070, P.R. China\nMay 6, 2026\nAbstract\nLarge language models are increasingly used as scientific agents, yet the flexibility that benefits\ngeneral-purpose agents can conflict with the accountability required in biomedical research. We\nstudy whether biomedical agents can be organized around auditable constraints rather than\nunconstrained autonomy. We presentOpen-Rosalind, a tool-first bio-agent system designed\naround four operational principles: evidence-grounded outputs, trace completeness, workflow-\nconstrained execution, and explicit tool mediation for factual claims. To evaluate these principles,\nwe introduceOpen-Rosalind BioBench, a process-aware benchmark that measures not only\ntask accuracy but also tool correctness, citation presence, trace completeness, and failure rate.\nOn a strict in-house benchmark, the reference pipeline achieves 81.4% accuracy with complete\nexecution traces. In multi-model ablations and paired replications, removing tools reduces\naccuracy by 19.3 to 26.4 percentage points, indicating that tool-first execution is the strongest\nand most stable contributor to performance. Constrained workflows also reduce lower-tail failures\nfor models that are weak at free-form tool use.\nHowever, an author-independent 30-task hold-out initially revealed severe external-validity\ncollapse on the deployment model. After diagnosing five routing and normalization failures\nand applying targeted fixes, hold-out accuracy improved from 17.8% to 53.3%, and the most\nconcerning negative comparison against ano_tool baseline disappeared. These results position\nOpen-Rosalind as a biomedical-agent study with an explicit external-validity audit, rather than\nas a claim that protocol constraints alone guarantee superior performance.\nKeywords: biomedical agents, large language models, auditability, tool-driven AI, scientific\nworkflows, evidence grounding, benchmarking\n1 Introduction\nWithin a span of two years, large language models have evolved from text completion engines into\nautonomous agents that plan, invoke external tools, and iterate over multi-step tasks. In domains\nsuch as software engineering and general information work, this transition has produced clear gains,\nin part because mistakes are rapidly visible and easily corrected. Biomedical and life-science research\nimpose a different standard. The problem is not capability: modern models can read a UniProt entry,\nparse a FASTA sequence, and reason about substitutions. The problem isepistemic accountability.\n1\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\nA scientific claim is not merely an assertion; it is a binding to a verifiable source. When a\ngeneral-purpose LLM agent confidently reports that a protein is located on a particular chromosome\narm, or cites a PubMed identifier in support of a mutation pathogenicity claim, the reader has no\noperational way to distinguish a tool-grounded fact from a hallucinated one. Worse, the agent may\nitself fail to draw this distinction, freely paraphrasing tool outputs alongside parametric knowledge\nin a single fluent paragraph. For wet-lab researchers, clinicians, or anyone whose downstream\ndecisions are guided by such answers, this opacity is a fundamental defect, not an implementation\ngap.\nThe position taken in this paper is that bio-agent systems require a different kind of design\ncontract than general-purpose agents:\nIn biomedical AI, the question is not whether an agent can produce an answer, but\nwhether the answer can be traced, audited, and replayed.\nPursuing this contract leads to four design constraints that shape every layer of the system:\nfactual content must come from explicit tools rather than parametric memory; every claim in\nan answer must be cited to a tool-derived source; every step of execution must be recorded in a\nreplayable trace; and the agent must operate within bounded, pre-declared workflows rather than\nfree-form planning loops. Each of these constraints sacrifices some flexibility in exchange for an\nauditable invariant.\nThis paper studiesOpen-Rosalind, a tool-first bio-agent system built around these constraints,\ntogether with a prototype system, a benchmark suite, and an external-validity repair study. We\nframe the work as three interlocking components. The first is aprincipled architecture(Sections 3–4)\nthat codifies the four design principles into a layered stack in which atomic tools, MCP-style\ntool interfaces, deterministic skills, a hybrid router, an agent, and a bounded multi-step harness\neach carry their own reproducibility invariant. The second is a prototype system (Section 5)\nthat demonstrates the architecture end-to-end on real biological tasks: sequence analysis, protein\nannotation, literature retrieval, and mutation assessment. The third isOpen-Rosalind BioBench\ntogether with controlled ablations and a hold-out audit (Sections6–7), designed not merely to\nmaximize raw task accuracy but to measure whether bio-agent systems satisfy the framework’s\nreproducibility guarantees and how those guarantees behave off-distribution.\nScope of claims. We do not argue that protocol discipline alone makes biomedical agents\nuniversally better, nor do we present Open-Rosalind as a finished product paper. The paper’s\nempirical claim is narrower. First, on the in-house benchmark, the tool-first design is a measurable\naccuracy lever relative tono_tool. Second, constraint appears to reduce lower-tail failures for\nmodels that are weak at unconstrained tool orchestration. Third, an external hold-out reveals that\nthese gains are conditional on routing robustness and tool coverage: before repairs, the system\ncollapses badly; after repairs, the most concerning negative comparison disappears, but a gap\nto ReAct remains. We therefore present the work as a biomedical-agent study with an honest\ndiagnosis-and-repair cycle, rather than as proof that constrained bio-agents dominate more flexible\nalternatives in every setting.\n2 From General Agents to Bio-Agents\nThe literature on LLM agents has converged on a small set of recurring patterns:ReAct-style\ninterleaved reasoning and acting,tool-use protocols, planner-executor decompositions, and recursive\ntask decomposition (e.g., AutoGPT-style loops). These patterns share a common philosophical\n2\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\ncommitment: flexibility is a virtue. The agent decides which tool to invoke, when to stop, when to\nbacktrack, and how to summarize. This commitment is well-suited to open-ended environments,\nwhere the cost of failure is low and the diversity of tasks resists pre-specification.\nBiomedical research is not such an environment. The set of legitimate workflows is small and\nwell-defined: annotate a protein, search the literature on a topic, evaluate the impact of a mutation,\ncompare two sequences. The cost of an unverifiable answer is high. The expectation that another\nresearcher can reproduce the analysis is not optional. Under these conditions, flexibility is not a\nvirtue but a liability: every additional degree of freedom afforded to the agent expands the surface\nover which errors and ambiguities can enter the result. Figure1 illustrates the key architectural\ndifferences between general-purpose agents and reproducible bio-agents.\nFigure 1: Architectural comparison: general-purpose agents vs. reproducible bio-agents. General\nagents prioritize flexibility with free-form reasoning and optional tool use; bio-agents prioritize\naccountability with mandatory tool-first execution, evidence grounding, and complete traceability.\nTable1 contrasts the design assumptions of general agents with those required for bio-agents.\nThe right column of Table1 should not be read as a list of capabilities omitted from general\nagents, but as a set ofguarantees added. A bio-agent gains less behavioral freedom than a general\nagent but offers stronger contracts: every fact has a source, every step is logged, every workflow\nterminates within a known bound. The remainder of this paper develops these guarantees into a\nconcrete framework.\n3\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\nTable 1: Design contrasts: general-purpose agents vs. reproducible bio-agents.\nDimension General-purpose agent Bio-agent (this work)\nKnowledge source LLM parametric memory + optional tools Tools only; LLM is a synthesizer\nPlanning Free-form, recursive, open-vocabulary Templated, bounded, closed-vocabulary\nTool invocation Optional, opportunistic Mandatory for all factual claims\nOutput format Natural language, untyped Typed evidence + cited summary\nFailure mode Best-effort answer, possible hallucination Explicit “insufficient evidence”\nTrace Best-effort logs (if any) Mandatory, replayable, structured\nStep bound None or LLM-decided Hard cap, externally enforced\nEvaluation Task accuracy Accuracy + tool correctness + evidence +\ntrace + failure rate\n3 Design Principles\nThe framework rests on four principles, summarized in Figure2. Each principle is operational — it\nis enforced by a specific component of the system — rather than aspirational.\n3.1 Tool-First Execution\nThe LLM never serves as a knowledge oracle. Every fact, computation, or annotation that appears in\nthe final answer must originate from an explicit tool invocation registered in the system. The LLM\nis invoked only after tools have produced structured outputs, and its role is reduced to synthesizing\nthose outputs into a natural-language summary. This re-positions the model as areader rather than\na source, and aligns the system with how scientific writing operates in practice: claims are made by\nciting data, not by recalling lore.\n3.2 Evidence-Grounded Outputs\nEvery factual claim in the LLM’s output must carry an inline citation to a tool-derived source —\na UniProt accession, a PubMed identifier, a structured statistic produced by an analysis routine.\nWhen the supporting evidence is insufficient, the LLM must say so explicitly rather than fall back on\nparametric guesses. This principle converts the conventional opacity of LLM outputs into something\ncloser to a scientific paragraph: each sentence is either bound to a citation or marked as a deliberate\nnon-answer.\n3.3 Trace Completeness\nEvery routing decision, every tool invocation, every observed return value, and the final LLM\nprompt are recorded in a structured, replayable trace. The trace is not a debugging aid but a\nfirst-class artifact, intended to support reproduction by external readers and auditing by reviewers.\nThe system treats a missing trace as a correctness failure on par with a wrong answer.\n3.4 Workflow-Constrained Execution\nTasks are executed through pre-declared workflows rather than free-form agent loops. Single-step\nqueries dispatch to one of a small fixed set of skills via a deterministic router; multi-step queries\n4\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\nFigure 2: The four operational principles of Open-Rosalind. Each principle constrains the LLM’s\nrole and is enforced by a specific layer of the architecture.\nexecute through one of a small fixed set of templated plans, under a hard step bound. The agent is\nnot permitted to invent new tools, restructure the plan mid-run, or recurse beyond the bound. This\nsacrifices flexibility deliberately: predictability and termination are valued above adaptability.\n4 Framework Architecture\nOpen-Rosalind organizes the four principles into a layered architecture in which each layer enforces\na specific reproducibility invariant. We describe the layers from the bottom up.\n4.1 Atomic Tools and MCP Interfaces\nAt the lowest level sit atomic tools — thin Python wrappers around external biological resources\n(UniProt, PubMed via Entrez) and local computations (BioPython sequence statistics, mutation\ndiff). Atomic tools are intentionally minimal: each performs a single operation, returns a structured\nobject, records latency and status, and never branches based on natural-language input. Tools are\nexposed to higher layers through MCP-style [1] interfaces, which provide a uniform contract for\ntool description, invocation, and result serialization. This boundary serves two purposes. First,\nit makes the system’s external dependencies explicit and version-controlled: a tool change is an\n5\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\ninterface change. Second, it allows alternative implementations (e.g., a local UniProt mirror, a\ncached PubMed index) to be substituted without changes to higher layers.\n4.2 Skills as Composed, Deterministic Pipelines\nA skill is the next layer up: a deterministic pipeline that composes one or more atomic tools\nto produce a typed output addressing a single biological question. The four skills used in this\nwork — sequence_basic_analysis, uniprot_lookup, literature_search, mutation_effect —\ncollectively cover the recurring questions in routine bioinformatics:what is this sequence?, what is\nthis protein?,what does the literature say?, what does this mutation do?.\nSkills are deliberately narrower than the LLM’s apparent reasoning capability. Their narrowness\nis the point: a skill is a unit of behavior that is small enough to test exhaustively, version explicitly,\nand reuse across both single-step and multi-step execution paths. Each skill carries a structured\nmanifest declaring its category, safety level (safe / network / compute), the atomic tools it depends\non, an input/output schema, and a small set of tested examples. Adding a new skill amounts to\ndeclaring its manifest and providing a handler; the rest of the framework discovers and integrates it\nwithout code changes.\n4.3 Hybrid Router\nThe router maps an incoming user query to a skill (single-step path) or to a multi-step plan\n(harness path). It is intentionally hybrid: a rule-based pre-filter handles the unambiguous cases —\na FASTA header, a UniProt accession pattern, a literature query containing the word “papers” —\nand an LLM-assisted intent classifier handles only the residual cases that mix natural language with\nembedded sequences or identifiers. The classifier is constrained to return a label from the small\nfixed skill set; it cannot invent new skills, and a parsing failure causes a deterministic fallback to\nthe rule-based result.\nThis hybrid design reflects a recurring tension in agent systems: rule-based routing is brittle but\nauditable, while LLM-based routing is flexible but opaque. Open-Rosalind resolves the tension by\ngiving rules first refusal: the LLM is consulted only when the rules cannot decide, and even then\nonly inside a closed-vocabulary contract.\n4.4 Single-Step Agent\nThe agent layer wraps the routed skill with the LLM-as-synthesizer step. Once the skill has executed\nand produced its typed output, the agent constructs an evidence-only prompt and asks the LLM to\nproduce a Markdown summary with inline citations. The agent enforces the evidence-grounded-\noutputs principle through prompt design (the system prompt forbids claims without citations)\nand through post-hoc validation (claims without recognizable evidence markers trigger a fallback\nanswer). The agent is also responsible for in-session context: a sliding window of prior turns is\nsupplied to the LLM, allowing follow-up questions such as “what species is this protein in?” to be\nresolved against the prior annotation.\n4.5 Multi-Step Harness\nThe harness orchestrates queries that require more than one skill. Rather than expose the agent to\narbitrary planning, the harness chooses from a small fixed set of templates — in the present imple-\nmentation: protein_research (sequence analysis→annotation→literature), literature_review\n(literature only), andmutation_assessment (mutation→annotation→literature). Each template\n6\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\nis a sequence of skill invocations with explicit slots for entities (e.g.,{protein_name}) that are\nfilled from prior step outputs.\nThe harness enforces three invariants. First, the planner cannot invent new templates; it can\nonly select from the registered set. Second, the runner observes a hard step bound (default five)\nthat terminates execution regardless of the plan. Third, the per-step trace is structurally identical\nto a single-step trace, so multi-step workflows are reproducible and auditable in the same way as\nsingle-step queries.\nThe combined data flow — query, mode selection, single-step or harness path, skill execution,\nevidence and trace generation, and final answer — ensures that every execution path respects the\nframework’s four principles.\n5 System Realization\nWe realized the proposed architecture as a compact experimental system consisting of a Python\nbackend, a React-based front-end, and an embedded SQLite store for sessions, messages, and traces.\nThe implementation is intentionally lightweight, because the scientific point is not scale but whether\nthe protocol can be enforced end-to-end without a heavyweight orchestration stack. Figure3\nsummarizes the resulting design.\nFigure 3: Open-Rosalind system architecture. The web UI communicates with a unified chat\nendpoint that dispatches to either single-step agent or multi-step harness paths. Both paths invoke\nskills from the registry, which in turn call atomic tools. All executions produce structured traces\nstored in SQLite.\n7\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\nThe model layer is exchangeable: the system communicates with any OpenAI-compatible\nendpoint (OpenRouter, vLLM, Azure, or on-premise inference servers). The experiments reported\nhere use google/gemma-4-26b-a4b-it via OpenRouter, but no part of the method is specific to\nthat model. Skills are loaded from declarative manifests at startup, and the front-end presents a\nunified chat interface whose execution path is chosen by the router.\nIn addition to the chat interface, the system records skill registry state, session history, and\nper-session traces in forms that can be inspected after the fact. These records are essential to the\ntrace-completeness principle: every interaction should leave behind material that can be reviewed,\nreplayed, and audited.\n6 Open-Rosalind BioBench\nConventional agent benchmarks measure task accuracy — the fraction of queries for which the final\nanswer is correct. For auditable bio-agents, accuracy is necessary but not sufficient: an answer\nthat happens to be right but is delivered through hallucinated tool calls, or with no recoverable\ntrace, still fails the framework’s contract.Open-Rosalind BioBenchis designed to make this\ndistinction empirically tractable.\n6.1 Splits and Provenance\nThe benchmark spans four task categories aligned with the four built-in skills: sequence analysis,\nprotein annotation, literature retrieval, and mutation assessment. The repository currently contains\n91 task instances across three splits:Basic (canonical inputs),Edge (mixed-modality or follow-up\ninputs), andMultiStep (queries requiring two or more skills). For the stricter model-comparison\nexperiments in Section7, follow-up variants are collapsed to 59 unique benchmark roots (32 Basic, 17\nEdge, 10 MultiStep) so that each root question contributes once to paired statistical tests. Tasks are\ndrawn from real biological objects — canonical UniProt accessions, PubMed-indexed publications,\nand proteins with documented variants — so that correctness can be checked against authoritative\nsources rather than model-generated ground truth.\n6.2 Metric Design\nFive metrics are reported per run.Task accuracymeasures whether the final answer satisfies the\ntask rubric. Tool correctnessmeasures whether the realized skill or harness template matches the\nexpected execution path. Citation presencemeasures whether factual claims carry recognizable\nevidence markers. Trace completenessmeasures whether the saved trace contains routing decisions,\ntool calls, statuses, and synthesis prompts.Failure ratemeasures crashes and timeouts separately\nfrom wrong answers. Because citation presence is partly syntactic, we additionally performed a\nmanual validity audit on 20 sampled responses: 19/20 had all cited claims materially supported by\nthe cited source, while 1/20 contained an over-broad paraphrase of a UniProt function field.\n6.3 Why a Process-Aware Benchmark\nThe benchmark is intended to separate two questions that are often conflated in agent papers:\nwhether a system returns the right answer, and whether it arrived there through a path that can be\naudited. The latter matters disproportionately in biomedical settings, where a wrong answer with a\ncomplete trace can be diagnosed and replayed, but a fluent answer with no trace cannot.\n8\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\n7 Empirical Results\n7.1 Internal Benchmark Under a Strict Scorer\nWe begin with experiments usinggoogle/gemma-4-26b-a4b-it. Earlier development dashboards\nscored the 91 turn-level tasks with a looser keyword-overlap rule and reported 100.0% / 93.9% /\n90.0% accuracy on Basic / Edge / MultiStep. For the paper’s main analyses, we instead use a\nstricter scorer on the 59 unique benchmark roots because it discriminates more sharply between\nconditions and supports paired statistical tests. The strict-scorer results are summarized in Table2.\nTable 2: Strict-scorer results on 59 unique BioBench roots using the reference Open-Rosalind\npipeline on Gemma. Trace completeness remains saturated, but accuracy drops materially on Edge\ninputs, making the benchmark more discriminative than the original loose dashboard scorer.\nSplit Tasks Accuracy Tool corr. CitePres Trace / No-fail\nBasic 32 87.5% 100.0% 3.1% 100.0% / 100.0%\nEdge 17 64.7% 76.5% 0.0% 100.0% / 88.2%\nMultiStep 10 90.0% 100.0% 0.0% 100.0% / 100.0%\nCombined 59 81.4% 93.2% 1.7% 100.0% / 96.6%\nTwo patterns matter. First, the Edge split is the real stress test: mixed natural language,\nembedded identifiers, and follow-up phrasing produce substantially lower accuracy than canonical\ninputs. Second, trace completeness remains saturated even when accuracy drops, meaning the\nframework localizes many failures to routing or factual synthesis rather than losing the execution\nrecord entirely. The weak point is citation presence under the strict per-sentence check: the current\nprompt tends to concentrate citations in a final evidence block rather than inline prose, so the\nmetric functions here as an architectural target rather than a saturated success number.\n7.2 Six-Model Ablations and Paired Replications\nTo test whether the framework’s behavior depends heavily on one underlying model, we ran a\nfive-condition ablation across six LLM families: the full Open-Rosalind pipeline, a free-form ReAct\nbaseline using the same atomic tools, ano_tool condition, ano_cite condition, and ano_template\ncondition. The aggregate results over 1,770 runs are summarized in Table3.\nTable 3: Aggregate six-model ablation over 59 unique tasks and five conditions (1,770 runs total).\nThe strongest and most stable effect is the drop fromfull to no_tool; citation and template\nablations change accountability properties much more than average accuracy.\nCondition Accuracy T ool corr. CitePres T raceComp NoF ail\nfull 81.4% 92.9% 0.3% 100.0% 96.3%\nreact 71.8% 86.2% 0.6% 82.5% 94.9%\nno_tool 57.3% 27.1% 0.3% 0.0% 100.0%\nno_cite 81.9% 93.2% 0.0% 100.0% 96.9%\nno_template 80.5% 93.2% 0.3% 100.0% 94.9%\n9\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\nFigure 4: Main comparison emphasized in the text. The deployment target (Gemma) and the most\nReAct-fragile model (GPT-5-mini) capture the paper’s two strongest claims:full substantially\noutperforms no_tool on both models, and it most clearly outperformsreact when unconstrained\ntool use is unstable.\nThe cleanest signal is the value of tools. Across models, removing tools collapses accuracy from\nroughly 80% to the mid-50s and drives tool correctness down to near-chance overlap with task\nkeywords. To test whether this effect survives a corrected unit of analysis, we ran paired seed\nreplications on the deployment target (Gemma) and on GPT-5-mini, which was the most ReAct-\nfragile model in the six-model sweep. Under cluster-aware permutation tests, thefull pipeline\nbeats no_tool by 26.4 percentage points on Gemma and 19.3 percentage points on GPT-5-mini\n(both p≤10−4). The comparison to ReAct is more nuanced. On GPT-5-mini, the constrained\npipeline beats ReAct by 35.3 percentage points and reduces catastrophic failures from 84.7% to\n39.0%. On Gemma, however,full and ReAct are statistically indistinguishable on the in-house\nbenchmark (p = 0.74). The support for the framework is therefore strongest as atool-first and\nlower-tail stabilityclaim, not as a universal claim that constrained workflows dominate free-form\nagents on raw accuracy.\n7.3 External Hold-Out, Diagnosis, and Repair\nTo probe external validity beyond the in-house benchmark, we generated a 30-task hold-out set\nauthored by an independent LLM and re-ran three paired conditions (full, react, no_tool) on\nGemma and GPT-5-mini. Table4 reports the Gemma results because Gemma is the deployment\ntarget and because the failure was most consequential there.\nThe initial hold-out materially changed the scientific interpretation of the system. On Gemma,\nthe constrained pipeline collapsed from internal-benchmark performance to 17.8% accuracy and\nwas significantly worse than both ReAct (∆ = −31.1 percentage points, p = 0.004) and no_tool\n(∆ = −28.9 percentage points, p = 0.002). Trace inspection showed that this was not a mysterious\nmodel failure. Most of the gap came from five concrete issues: stop-words such as “information” and\n10\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\nFigure 5: Experimental results beyond the architectural diagrams. Left: accuracy offull, react,\nand no_tool across six model families, showing that the constrained pipeline most clearly improves\nthe lower-performing tool-use settings. Right: process-aware benchmark metrics aggregated across\nall models and conditions, illustrating thatno_tool collapses tool correctness and trace completeness\neven when it remains failure-free.\nTable 4: Gemma results on the 30-task author-independent hold-out before and after the rout-\ning/normalization repairs. The most important change is thatfull is no longer significantly worse\nthan no_tool after the fixes, although it remains below ReAct.\nSetting full react no_tool\nBefore fixes 17.8% 48.9% 46.7%\nAfter fixes 53.3% 70.0% 57.8%\n“characterize” being preserved in search queries; natural-language mutation forms such as “KRAS\nG12D” failing to trigger the mutation workflow; Greek-vs-ASCII mismatches in keyword matching;\nembedded sequences going undetected when wrapped in prose; and insufficient coverage of literature\nkeywords such as “publications” or “studies”.\nAfter targeted fixes to the router, query cleaning, and scorer normalization, Gemma improved by\n35.5 percentage points to 53.3%. More importantly, the most concerning negative comparison disap-\npeared: full vs. no_tool became statistically indistinguishable (p = 0.72, 95% CI[−23.3,+13.3]).\nThe gap to ReAct narrowed but remained significant (∆ =−16.7 percentage points,p = 0.032). This\nrepair cycle clarifies the right interpretation of Open-Rosalind. Routing robustness isnecessary for\nexternal validity, and several of the original failures were patchable engineering faults. The remaining\ngap, however, reflects a real design tradeoff: a tool-first system refuses to invent functional claims\nthat its current tools cannot support. We therefore treat the hold-out not as an embarrassment to\nhide, but as evidence that bio-agent papers should report diagnosis-and-repair cycles rather than\nonly polished in-distribution numbers.\n8 Discussion\nPosition relative to general-purpose agent frameworks.ReAct [20] and its descendants\nassume that flexibility is intrinsically valuable. For research domains in which the set of legitimate\n11\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\nFigure 6: Gemma hold-out accuracy before and after the routing and normalization repairs. The\nmost important change is not simply the gain infull accuracy, but the disappearance of the\nstrongly negative full vs. no_tool comparison that had undermined the original claim.\ntasks is small, the cost of incorrect outputs is high, and reproducibility is non-negotiable, this\nassumption inverts. Open-Rosalind is in part an argument that the bio-agent literature should\nadopt a different default: that flexibility be unlocked deliberately, and that every degree of freedom\ngranted to the agent be paid for in additional verification machinery.\nPosition relative to biomedical LLMs. A complementary line of work — BioGPT [7],\nBioMedLM [6], Med-PaLM [5] — improves the LLM’s biomedical knowledge through special-\nized pre-training. Open-Rosalind is orthogonal to this line: the framework seeks to restrict what a\nmodel may assert without evidence. The experiments suggest that a disciplined protocol can raise\nthe performance floor of smaller models, but they do not imply that protocol alone substitutes for\nmodel capability in every regime.\nPosition relative to recent bio-agent systems.The past year has seen rapid progress in\nLLM-based biomedical agents.Biomni [8] is a general-purpose biomedical AI agent that unifies\nreasoning, retrieval, and code execution across diverse biological domains (genomics, immunology,\nmicrobiology, neuroscience), emphasizing breadth and autonomous task execution through free-form\nplanning. GeneAgent [9] addresses hallucination in gene-set analysis by autonomously interacting\nwith biological databases to verify its own output, introducing a self-verification loop that reduces\nparametric recall errors. STELLA [10] employs a multi-agent architecture with an evolving\ntemplate library and dynamic tool ocean, enabling self-improvement through iterative refinement. A\nrecent comprehensive review [\n11] surveys the landscape of LLM agents for biomedicine, highlighting\nmethods, evaluations, and challenges across clinical and research applications.\nConcurrent work has also addressed reproducibility and evaluation.R-LAM [12] proposes\nreproducibility-constrained large action models for scientific workflow automation, introducing\nstructured action schemas and deterministic execution traces. Several benchmarks have emerged: an\nAI Agent Evaluation Suite for Bioinformatics[13] evaluates frontier models across multiple\n12\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\nagent harnesses using LLM-based graders; aScientific Research Suite[14] provides rigorous\nbenchmarking for AI agents in scientific contexts; and frameworks formulti-agent scientific AI\nevaluation [15] propose process-level metrics, scaling curves, and adversarial robustness tests. A\nNature Methods perspective [16] surveys the broader landscape, noting the transformative potential\nof autonomous systems while emphasizing the need for verification mechanisms.\nOpen-Rosalind’s distinct contribution relative to these systems is its focus onprotocol discipline\nas a first-class design constraint. Where Biomni and STELLA prioritize capability breadth and\nself-evolution, and GeneAgent prioritizes self-verification, Open-Rosalind prioritizesreproducibility\nguarantees enforced at every architectural layer: every fact must originate from a tool, every step\nmust be logged, every workflow must terminate within a known bound. The hold-out repair cycle in\nSection 7.3 shows both the value and the limit of this position. These guarantees are not automatic\n— they require careful routing robustness and enough tool coverage to support the requested claims.\nWe therefore view recent bio-agent systems as complementary: some explore what autonomous\nbio-agentscan do; Open-Rosalind asks what theymust guaranteeto be trusted in scientific workflows.\nLimitations. The benchmark is still small and partly internal: the 91 task instances and 59 unique\nbenchmark roots were authored during development, and the external hold-out, while independent of\nthe system author, was still LLM-authored rather than human-authored. The harness templates are\nhand-coded; while their fixity is a feature for reproducibility, it bounds the range of workflows the\nsystem can express. The literature skill depends on PubMed availability and abstract coverage, and\nthe mutation skill currently supports sequence-grounded assessment more strongly than mechanistic\nfunctional prediction. Finally, the hold-out results show that external validity remains only partially\nrecovered: after repairs, the most worrying negative comparison disappears, but the full-vs-ReAct\ngap remains on Gemma.\nFuture work. Three directions are immediate. First, expanding tool coverage for the failure\nclasses exposed by the hold-out: richer mutation-effect predictors, broader literature retrieval, and\nontology-backed protein annotation fields. Second, replacing the current internal benchmark with a\nhuman-authored blind split and multi-rater scoring protocol. Third, pushing reproducibility one\nstep further by caching external tool responses and pinning model versions more tightly, so that\ntraces become not only auditable but closer to bit-level replayable.\n9 Conclusion\nOpen-Rosalind shifts the design of bio-agent systems from free-form reasoning to bounded, traceable\nworkflow execution. Its main contribution is best read as a biomedical-agent study with an honest\nexternal-validity audit: a layered architecture with explicit invariants, a concrete experimental\nrealization, a process-aware benchmark, and a diagnosis-and-repair cycle that makes the limits of\nthe approach visible rather than implicit. The internal experiments support tool-first execution as a\nreal accuracy lever, especially againstno_tool baselines, and support constrained workflows as a\nlower-tail stabilizer for models that are weak at free-form tool use. The external hold-out, however,\nshows that these advantages are conditional on routing robustness and tool coverage. We hope the\nframework, the benchmark, and the released traces are useful not only as research artifacts, but as a\nconcrete template for how biomedical agent papers can make their accountability claims measurable.\n13\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\nCode, Data, and Reproducibility\nThe system, benchmark tasks, hold-out data, and specification are released open-source under\nthe MIT license at https://github.com/maris205/open-rosalind. Per-task traces from the\nexperiments reported here are included in the repository. The current release supports strong\nauditability through saved routing decisions, tool I/O, statuses, and synthesis prompts; exact replay\nstill depends on live external APIs and hosted model versions, which we list as next-step work rather\nthan a solved property.\nReferences\n[1] Anthropic. Model context protocol (MCP) specification.https://modelcontextprotocol.io,\n2024.\n[2] Anthropic. Build with Claude Skills. https://docs.claude.com/en/docs/\nbuild-with-claude/skills, 2025.\n[3] ByteDance. DeerFlow: A modular agent skill framework.https://github.com/bytedance/\ndeerflow, 2025.\n[4] OpenBioMed Team. OpenBioMed: An open multimodal biomedical foundation framework.\n2025.\n[5] K. Singhal, T. Tu, J. Gottweis, et al. Towards expert-level medical question answering with\nlarge language models.arXiv preprint arXiv:2305.09617, 2023.\n[6] Stanford CRFM. BioMedLM: A 2.7B parameter language model trained on biomedical text.\n2023.\n[7] R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T.-Y. Liu. BioGPT: Generative\npre-trained transformer for biomedical text generation and mining.Briefings in Bioinformatics,\n23(6):bbac409, 2022.\n[8] K. Huang, et al. Biomni: A general-purpose biomedical AI agent.bioRxiv, 2025.05.30.656746,\n2025.\n[9] Y. Huang, et al. GeneAgent: Self-verification language agent for gene-set analysis using domain\ndatabases. Nature Methods, 2025.https://doi.org/10.1038/s41592-025-02748-6.\n[10] Self-Evolving LLM Agent for Biomedical Research.arXiv preprint arXiv:2507.02004, 2025.\n[11] Large Language Model Agents for Biomedicine: A Comprehensive Review of Methods,\nEvaluations, Challenges, and Future Directions. Information, 16(10):894, 2025. https:\n//www.mdpi.com/2078-2489/16/10/894.\n[12] Reproducibility-Constrained Large Action Models for Scientific Workflow Automation.arXiv\npreprint arXiv:2601.09749, 2025.\n[13] An AI Agent Evaluation Suite for Bioinformatics.arXiv preprint arXiv:2601.21800, 2025.\n[14] Rigorous Benchmarking of AI Agents with a Scientific Research Suite. arXiv preprint\narXiv:2510.21652, 2025.\n14\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint \n\n[15] Toward Evaluation Frameworks for Multi-Agent Scientific AI Systems. arXiv preprint\narXiv:2603.26718, 2026.\n[16] Nature Methods Editorial. Artificial intelligence agents for biology.Nature Methods, 2025.\nhttps://doi.org/10.1038/s41592-025-02958-y.\n[17] The UniProt Consortium. UniProt: the universal protein knowledgebase in 2023.Nucleic Acids\nResearch, 51(D1):D523–D531, 2023.\n[18] National Center for Biotechnology Information. Entrez programming utilities help.https:\n//www.ncbi.nlm.nih.gov/books/NBK25501/, 2024.\n[19] P. J. A. Cock, T. Antao, J. T. Chang, et al. Biopython: Freely available Python tools for\ncomputational molecular biology and bioinformatics.Bioinformatics, 25(11):1422–1423, 2009.\n[20] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. ReAct: Synergizing\nreasoning and acting in language models. InICLR, 2023.\n[21] L. Wang. Emergence of biological structural discovery in general-purpose language models.\nbioRxiv, 2026.01.03.697478, 2026.\n15\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted May 8, 2026. ; https://doi.org/10.64898/2026.05.06.722404doi: bioRxiv preprint","source_license":"CC-BY-4.0","license_restricted":false}