Instruction Strategy Design for Autonomous Machine Learning Experimentation Systems: A Taxonomy, Cross-System Analysis, and Evidence-Based Practitioner Framework

Instruction Strategy Design for Autonomous Machine Learning Experimentation Systems: A Taxonomy, Cross-System Analysis, and Evidence-Based Practitioner Framework | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Instruction Strategy Design for Autonomous Machine Learning Experimentation Systems: A Taxonomy, Cross-System Analysis, and Evidence-Based Practitioner Framework Praneeth Kodumagulla This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9286871/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 13 You are reading this latest preprint version Abstract Autonomous machine learning experimentation systems—wherein a large language model (LLM) agent iteratively proposes, executes, and evaluates code modifications against a fixed scalar metric—represent a fundamental shift in how machine learning research is conducted. In these systems, the practitioner's primary lever is not the training code itself but the natural-language research program : the instruction document that specifies objectives, priorities, and constraints for the agent across dozens or hundreds of consecutive decisions. Despite this centrality, no principled framework for designing research programs exists in the literature. This survey addresses that gap through four contributions. First, we conduct a structured cross-system analysis of sixteen agentic AutoML and autonomous research systems—including AIDE, AIRA, R&D-Agent, AgentHPO, AlphaEvolve, MLAgentBench, AI-Researcher, and AI Scientist-v2—identifying the instruction document as a universal practitioner-facing control mechanism and cataloguing seven design dimensions. Second, we develop a five-family taxonomy of instruction strategies: Scope-Constrained, Hypothesis-Directed, Diversity-Preserving, Simplicity-Biased, and Curriculum-Staged, grounded in theory from the AutoML, evolutionary computation, prompt engineering, and curriculum learning literatures. Third, we provide multi-source empirical grounding: analysis of two publicly documented overnight sessions suggests a cross-session curriculum intervention is associated with a 37% difference in total gain, with important caveats regarding session-length confounding; independently controlled benchmarks from AIRA and AgentHPO corroborate the taxonomy's predictions. Fourth, five practitioner guidelines with explicitly labelled calibration thresholds are synthesised and validated against all sixteen surveyed systems. autonomous research agents agentic AutoML research program design instruction engineering LLM-guided optimisation 1 Introduction The practice of machine learning (ML) research has long been bounded by a fundamental throughput constraint: the human researcher. Each experimental cycle—formulate a hypothesis, write and execute code, interpret metrics, decide whether to proceed—requires skilled judgment at every step, with wall-clock times measured in hours. Even in well-resourced settings, a single researcher executes only a handful of substantive experiments per day, making this the binding limit on how quickly the community can explore model configurations, architectures, and training strategies. The emergence of LLMs capable of reading, writing, and reasoning about code has created a qualitatively different possibility. By delegating the operational layer to an autonomous agent, the human researcher's role shifts from direct experimentation to strategic direction-setting. An overnight session on a single GPU can execute 80–100 experiments; distributed multi-agent deployments can cover hundreds more, representing throughput improvements of roughly two orders of magnitude over manual iteration (Karpathy 2026 ; Toledo et al. 2025 ). The significance of this shift is reflected in the performance of current systems on the MLE-bench benchmark (Chan et al. 2024 )—75 real-world Kaggle ML engineering competitions: AIDE with o1-preview achieves a medal rate of 16.9%, R&D-Agent achieves 35.1%, and the best current open-source system achieves a score that would have been inconceivable without autonomous experimentation. Yet despite this progress, the instruction documents that direct these systems remain entirely unanalysed as an engineering artefact. This paradigm sits at the intersection of two active research frontiers. The first is LLM-driven automated algorithm design and meta-optimisation—including LLaMEA, which uses GPT to generate and iteratively evolve complete metaheuristic algorithm code (van Stein and Bäck 2024 ), and LLM-driven meta-optimisers for automated intelligent optimisation more broadly (Zheng et al. 2026 ). The second is the emergence of production-scale autonomous ML research systems: autoresearch (Karpathy 2026 ), which accumulated more than 50,000 GitHub stars within three weeks of release; AIDE (Jiang et al. 2025 ); AIRA (Toledo et al. 2025 ); R&D-Agent (Yang et al. 2025 ); AgentHPO (Liu et al. 2024a ); AlphaEvolve (Novikov et al. 2025 ); and MLAgentBench (Huang et al. 2024 )—which established the foundational benchmark for evaluating LLM agents on ML experimentation tasks (13 tasks, best agent 37.5% success rate with Claude 3 Opus), predating and motivating the current generation of systems. Across all these systems, one structural feature is universal and entirely unstudied: the natural-language instruction document that is the sole artefact under deliberate human control once a session begins. In autoresearch this is program.md; in AIDE it is the task specification; in AIRS-Bench (Lupidi et al. 2026 ) it is project_description.md; in MLAgentBench (Huang et al. 2024 ) it is the task description. This document determines what the agent considers a valid modification, how it balances exploration and exploitation, what prior knowledge it carries from previous sessions, and how it behaves when progress stalls. Yet the design of this document has received almost no analytical attention in the literature. The contrast with related work in automated algorithm configuration (Bäck et al. 2023 ) and LLM-based heuristic design (van Stein and Bäck 2024 ; van Stein et al. 2026 ) is striking: those domains have rich theoretical frameworks for design-space navigation; the analogous problem for autonomous ML research programs does not. The prompt engineering literature establishes that LLM behaviour is strongly sensitive to instruction wording even in single-turn inference tasks—with documented performance differences of up to 76 accuracy points across semantically equivalent prompt formats (Sclar et al. 2024 ). In an autonomous experimentation session we hypothesise that these effects compound: each agent decision is conditioned on prior decisions and their outcomes, so an instruction that biases the agent toward unproductive territory in experiment 10 also shapes the context for experiments 11 through 100. This compounding hypothesis is supported by AIRA's finding (Toledo et al. 2025 ) that search-policy performance differences widen with session length—a prediction directly analogous to the session-length sensitivity observed in hyperparameter optimisation (Snoek et al. 2012 ). This paper addresses the gap through a structured survey and taxonomy. We survey sixteen agentic AutoML and autonomous research systems, extract the common role and design dimensions of their instruction documents, develop a five-family taxonomy grounded in theory and empirical evidence, validate the taxonomy across systems, and synthesise practitioner guidelines. Contributions. Four specific contributions are made: (1) A structured cross-system survey of sixteen agentic AutoML and autonomous research systems, identifying the instruction document as a universal practitioner-facing control mechanism and cataloguing seven design dimensions. (2) A five-family taxonomy of instruction strategies grounded in theory from the AutoML, evolutionary computation, prompt engineering, and curriculum learning literatures, with each family characterised along four analytical dimensions. (3) Multi-source empirical grounding drawn from publicly documented session logs, quantitative benchmarks from AIRA and AgentHPO under independently controlled conditions, and community-reported distributed observations; cross-validation of each family against at least two independent systems. (4) Five practitioner guidelines with explicitly labelled calibration thresholds, a validation matrix, and a structured research agenda for future controlled study. The paper is organised as follows. Section 2 reviews related work and positions the paper with respect to existing surveys. Section 3 frames three research questions. Section 4 surveys the landscape. Section 5 develops the taxonomy. Section 6 provides empirical grounding. Section 7 derives practitioner guidelines. Section 8 discusses limitations and future directions. Section 9 concludes. 2 Related Work 2.1 Surveys of Agentic AutoML and Autonomous Research Systems Several survey papers have addressed the broader landscape of LLM-based autonomous agents in scientific research and machine learning engineering. Wang et al. ( 2024 ) provide a comprehensive survey of LLM-based autonomous agents, establishing a unified framework covering memory, planning, action, and tool use—but their survey addresses general-purpose agents rather than the specific class of systems that autonomously modify and evaluate code within a fixed evaluation loop. Gardner et al. ( 2026 ) review LLMs as autonomous agents and tool users with a focus on the role of tools and feedback loops in agentic behaviour; instruction document design falls outside their scope. The Agentic AI survey (Plaat et al. 2025 ) provides a taxonomy of agentic architectures across symbolic and neural paradigms but does not address the design of the practitioner-facing control artefact. The AutoML literature review (Baratchi et al. 2024 ) surveys classical AutoML methods but predates the LLM-driven autonomous experimentation paradigm introduced by AIDE, AIRA, and autoresearch. The present paper occupies the gap between these surveys: it addresses the specific class of LLM-driven autonomous ML experimentation systems and, uniquely, focuses on the practitioner-facing instruction document rather than the agent architecture. 2.2 Prompt Engineering and Instruction Quality The prompt engineering literature establishes sensitivity of LLM outputs to instruction wording (Sclar et al. 2024 ; Renduchintala et al. 2024 ; Zhuo et al. 2024 ) and provides gradient-free methods for single-turn prompt optimisation (Li et al. 2025 ). However, this literature addresses single-turn inference tasks—where each prompt is evaluated independently—whereas research program design addresses multi-turn, state-accumulating experimentation sessions where each agent decision conditions all subsequent ones. LLaMEA (van Stein and Bäck 2024 ) and the in-the-loop HPO work (van Stein et al. 2026 ) address LLM-driven algorithm design in the evolutionary computation tradition; their focus is on the algorithm text rather than the practitioner-facing program specification. The present paper bridges these communities: it applies the analytical rigour of prompt sensitivity research to the multi-turn, session-level instruction design problem in autonomous ML research. 2.3 Benchmarks for Autonomous ML Experimentation MLAgentBench (Huang et al. 2024 ) was the first systematic benchmark for evaluating LLM agents on ML experimentation, introducing 13 tasks and establishing that the best agent (Claude 3 Opus) achieves a 37.5% average task success rate but struggles with long-term planning. MLE-bench (Chan et al. 2024 ) extended this to 75 real-world Kaggle competitions, finding that the best agent setup (AIDE with o1-preview) achieves a 16.9% medal rate at pass@1. Both benchmarks treat the instruction document as a fixed given rather than as a variable to be optimised. AIRS-Bench (Lupidi et al. 2026 ) provides a 20-task benchmark where project_description.md files are the primary control variable, making it the closest precedent to the present paper's focus; however, it does not provide a framework for designing these documents. The present survey and taxonomy provide that missing design framework. 3 Research Questions This survey is organised around three explicit research questions, following the methodological convention for structured literature reviews (Kitchenham and Charters 2007 ; Petersen et al. 2008 ): RQ1. What is the role of instruction documents in current agentic AutoML and autonomous research systems, and what design dimensions do they expose? RQ2. What are the theoretically distinct families of instruction strategies, and what do they predict about agent search behaviour, exploration–exploitation balance, and session-level outcomes? RQ3. What empirical evidence supports the predicted effects of different instruction strategies, and how does this evidence validate the taxonomy across systems? These questions collectively address the absence of a principled framework for instruction document design in autonomous experimentation systems. 4 Survey of Autonomous Research and Agentic AutoML Systems 4.1 Survey Methodology We surveyed systems published or released between 2023 and March 2026 satisfying three criteria: (a) an LLM agent autonomously modifies code or experimental configurations; (b) modifications are evaluated against a scalar metric without human intervention; and (c) the agent iterates over multiple cycles within a session. Systems were identified through arXiv searches on 'autonomous ML research', 'agentic AutoML', 'LLM-driven code optimisation', and 'AI research agent', supplemented by forward citation tracking from AIDE (Jiang et al. 2025 ) and backward citation tracking from AIRS-Bench (Lupidi et al. 2026 ), and by reviewing related-work sections of AIRA (Toledo et al. 2025 ), R&D-Agent (Yang et al. 2025 ), and MLAgentBench (Huang et al. 2024 ). Sixteen systems satisfied all criteria. Taxonomy development by a single author lacks inter-rater reliability; we address this by requiring each strategy family to be independently corroborated by at least two systems beyond the anchoring case (Sect. 6.3 ). Saturation was assessed by checking whether the three systems added in the final pass (R&D-Agent, MLGym, AutoKaggle) introduced design choices not representable by the seven-dimension framework; they did not. Table 1 summarises results. Table 1 Cross-system survey of instruction documents in agentic AutoML and autonomous research systems (16 systems, 2023–March 2026) System Instruction Document Key Design Dimensions Search Mechanism Evaluation Metric autoresearch(Karpathy 2026 ) program.md (Markdown) All 7 dimensions (Table 2 ); richest across systems Greedy hill-climbing val_bpb AIDE(Jiang et al. 2025 ) Task specification (text) Scope, hypothesis hints, safety constraints Tree search: Draft/Improve/Debug MLE-bench medal rate (16.9% o1-preview) AIRA(Toledo et al. 2025 ) AIRA-dojo task spec (YAML+text) Scope, operator set, search policy, exploration budget Configurable: Greedy/MCTS/Evolutionary MLE-bench medal rate R&D-Agent(Yang et al. 2025 ) Research task brief (text) Scope, hypothesis seeding, exploration directive Dual-agent: Researcher + Developer, parallel traces MLE-bench medal rate (35.1% — SOTA) AgentHPO(Liu et al. 2024a ) HPO task description + rationales Hypothesis list, mechanistic rationales, prior experiment summaries Sequential LLM Bayesian-style reasoning Task accuracy/RMSE (12 benchmarks) MLAgentBench(Huang et al. 2024 ) Task description (text + starter files) Task objectives, allowed operations, evaluation protocol ReAct-style: read/write/execute code Task success rate (37.5% Claude Opus — SOTA at time) AlphaEvolve(Novikov et al. 2025 ) Problem specification (text) Scope, correctness requirements, diversity hints Evolutionary: Flash (volume) + Pro (depth) Correctness + efficiency AI-Researcher(Tang et al. 2025 ) Research direction brief Domain scope, resource filters, idea evaluation criteria Literature-driven idea generation pipeline Novelty + feasibility scores AI Scientist(Lu et al. 2024 ) Research topic + templates Topic constraints, template, review criteria Full pipeline: ideas→experiments→paper Reviewer score simulation AI Scientist-v2(Yamada et al. 2025 ) Workshop task description Research scope, milestone structure, collaboration roles Agentic tree search + sub-agents Workshop acceptance criteria AIRS-Bench(Lupidi et al. 2026 ) project_description.md (per task) Scope, baseline info, allowed modifications, metric target AIRA-dojo or MLGym harness Normalised score vs. SOTA Agent Lab(Schmidgall et al. 2025 ) Research task specification Literature scope, experiment plan, sub-agent roles Multi-agent literature + experiment pipeline Benchmark performance SELA(Chi et al. 2024 ) ML task description Pipeline components, search depth MCTS over full AutoML pipeline Validation accuracy FunSearch(Romera-Paredes et al. 2023 ) Function spec (skeleton + docstring) Function interface, correctness, diversity targets Evolutionary: LLM mutation + evaluator Combinatorial objective AutoKaggle(Li et al. 2024b ) Competition description (parsed) Feature scope, model selection, validation strategy Multi-agent data science pipeline Kaggle leaderboard score MLGym(Nathani et al. 2025 ) Task description file (text + code) Research objectives, allowed methods, evaluation protocol, baseline reference ReAct-style LLM agent; RL-trainable Task-specific benchmark score 4.2 Design Dimensions of Instruction Documents Across all sixteen systems, the instruction document translates the practitioner's intent, knowledge, and constraints into agent-actionable form. Seven design dimensions appear consistently. Table 2 summarises these dimensions, their presence, and the range of choices observed. Table 2 Design dimensions of instruction documents across sixteen surveyed systems. Presence: H = high (explicitly supported), M = moderate (partially supported), L = low (implicit or absent) Design Dimension Description and Range of Choices Presence (16 Systems) Modification scope Which components the agent may alter. Range: unconstrained to component-specific with explicit exclusion zones. H in all 16 Hypothesis seeding Whether starting hypotheses are provided and with what supporting detail. Range: no seeding to ranked hypothesis trees with mechanistic rationales. H in 10; M in 6 Simplicity criterion Whether complexity carries a penalty alongside metric performance. Range: pure metric maximisation to hard complexity veto above threshold. H in 5; M in 4; L in 7 Cross-session memory Whether prior session findings are injected into the current instruction document. Range: none to full injection of prior discoveries as starting configurations. H in 3; M in 6; L in 7 Exploration directive Explicit instruction on exploration–exploitation balance. Range: implicit to hard category cap per N experiments with explicit exploration budget per phase. H in 5; M in 6; L in 5 Curriculum staging Whether the document specifies different strategies for different session phases. Range: uniform throughout to three-phase with convergence-triggered transitions. H in 3; M in 4; L in 9 Safety constraints Hard limits protecting session integrity. Range: soft resource limit only to hard parameter bounds with architectural invariants. H in 8; M in 5; L in 3 4.3 The autoresearch Framework as Anchoring Case Among the sixteen surveyed systems, autoresearch (Karpathy 2026 ) exposes the richest set of design dimensions and its minimalist architecture makes program.md effects maximally visible. The system consists of a single approximately 630-line training script (train.py) modified freely by the agent; an immutable evaluation harness (prepare.py); and a human-authored Markdown instruction document (program.md). Every training run is fixed at exactly five minutes of wall time, making val_bpb—validation bits-per-byte, a vocabulary-size-independent language modelling metric—directly comparable across all experiments. On a single NVIDIA H100 GPU, approximately 12 experiments per hour are achievable, yielding approximately 100 experiments during a typical overnight session. The agent operates in a greedy hill-climbing loop: read program.md, propose a modification to train.py, execute the run, commit if val_bpb improved, revert if not. Because the agent has no other persistent state and every experimental outcome is numerically comparable, differences in session-level performance between program variants can be attributed to instruction design with unusually high confidence relative to more complex multi-component systems. 4.4 Answer to RQ1 Across all sixteen surveyed systems, the instruction document is the practitioner's primary control mechanism. Seven design dimensions are common, though present to varying degrees. The design space is large, combinatorial, and currently navigated ad hoc. The taxonomy in Sect. 5 characterises the theoretically distinct strategy families that emerge from different configurations of these dimensions. 5 Taxonomy of Instruction Strategy Families 5.1 Taxonomy Development Method The taxonomy was developed through structured content analysis of five inputs: (1) the reference autoresearch program.md and all publicly documented session variants (Karpathy 2026 ); (2) the cross-system survey in Sect. 4 ; (3) community fork activity and program variants in the autoresearch GitHub Discussions; (4) design choices documented in AIDE, AIRA, R&D-Agent, AgentHPO, AlphaEvolve, MLAgentBench, AI-Researcher, and AIRS-Bench; and (5) theoretical frameworks from the prompt engineering (Sclar et al. 2024 ; Renduchintala et al. 2024 ; Zhuo et al. 2024 ), AutoML (Snoek et al. 2012 ; Wistuba et al. 2015 ), evolutionary computation (Bäck et al. 2023 ; van Stein and Bäck 2024 ), curriculum learning (Bengio et al. 2009 ), and population-based training (Jaderberg et al. 2017 ) literatures. A strategy family was defined as a coherent configuration of dimension choices that: (a) appears repeatedly across sources; (b) is theoretically distinguishable from other families along at least two analytical dimensions; and (c) has direct empirical support or a clear theoretical prediction from the related literature. Five families resulted. They are not mutually exclusive—effective programs typically combine elements of two or three families. Each family is characterised along four dimensions: (i) exploration–exploitation balance; (ii) modification emphasis; (iii) primary failure mode; and (iv) conditions of applicability. Table 3 summarises all five families. 5.2 Scope-Constrained Strategy A Scope-Constrained program explicitly restricts permissible modifications to a specified system component—the attention mechanism, the optimiser, positional encoding, or any other named subsystem. The rationale is search efficiency through dimensionality reduction: constraining scope increases the density of productive experiments per unit of compute, directly analogous to search space pruning in classical AutoML (Wistuba et al. 2015 ). Where classical pruning is algorithmic, scope-constraining is linguistic. In AIRA terminology (Toledo et al. 2025 ), this corresponds to restricting the operator set to a component-specific subset. In AIRS-Bench (Lupidi et al. 2026 ), the project_description.md files instantiate this pattern. In R&D-Agent (Yang et al. 2025 ), the research task brief scopes the Researcher agent's hypothesis space. In MLAgentBench (Huang et al. 2024 ), task descriptions constrain the allowable code operations. Exploration–exploitation balance. Strongly exploitative; all session compute is directed toward depth within the constrained component. Modification emphasis. Component-specific architectural adjustments or hyperparameter tuning within the named subsystem. Primary failure mode. Premature convergence if the constrained component lacks optimisation headroom. Ill-suited to early exploratory sessions where the productive region is unknown. Conditions of applicability. Most effective when prior evidence establishes that a specific component is under-optimised, or as the second phase of a curriculum-staged session. 5.3 Hypothesis-Directed Strategy A Hypothesis-Directed program provides structured testable hypotheses, each paired with a mechanistic rationale explaining the expected effect under the session's specific constraints. For example: "Halving total batch size will increase gradient step count within the 5-minute budget by approximately 40%; the training signal remains stable above approximately 32K tokens per step at this model scale, so the additional gradient steps should translate to improved val_bpb." Pairing rationales with hypotheses engages the agent's chain-of-thought reasoning capabilities (Wei et al. 2022 ), improving the quality of code modifications. AgentHPO (Liu et al. 2024a ) validates this strategy under controlled conditions. Across twelve benchmark HPO tasks, agents provided with mechanistic rationales reached competitive performance with fewer function evaluations than baseline agents operating without structured scaffolding—the paper reports improvements across all twelve tasks, with the hypothesis-directed agent requiring roughly half the number of trials to match the best configurations found by the baseline in the median case. R&D-Agent (Yang et al. 2025 ) independently validates this at system level: its explicit researcher–developer separation, where the Researcher agent constructs mechanistic hypotheses before implementation, is credited as the key driver of its state-of-the-art 35.1% MLE-bench medal rate, compared to 16.9% for AIDE without this mechanism. DS-Agent (Guo et al. 2024 ) provides further corroboration via case-based reasoning provision. Exploration–exploitation balance. Focused exploitation of practitioner domain knowledge. The instruction should authorise the agent to generate its own hypotheses as the list is exhausted, preventing anchoring bias. Modification emphasis. Hypothesis-driven modifications within the components addressed by the provided rationales. Primary failure mode. Anchoring bias—excessive concentration on the provided list even when early experimental evidence indicates those hypotheses are unproductive. Conditions of applicability. Most effective for refinement sessions where the practitioner has strong domain theory about which modifications are likely to improve the metric and why. 5.4 Diversity-Preserving Strategy A Diversity-Preserving program explicitly instructs the agent to maintain variety across modification categories—capping consecutive experiments within any single category and mandating rotation across predefined categories. This counteracts the hyperparameter-fixation failure mode of greedy hill-climbing, which is structurally analogous to exploration collapse in RL-based agent self-evolution (Wang et al. 2025 ). The population-based training literature (Jaderberg et al. 2017 ) provides the theoretical foundation. This principle is directly connected to diversity maintenance in evolutionary algorithms (Bäck et al. 2023 )—population diversity in evolutionary computation corresponds to temporal modification diversity in a single-agent session. AlphaEvolve (Novikov et al. 2025 ) instantiates this at system level through its dual-model architecture (Gemini Flash for high-volume exploration, Gemini Pro for selective refinement); FunSearch (Romera-Paredes et al. 2023 ) maintains island-based populations with explicit diversity preservation. MLAgentBench (Huang et al. 2024 ) found that agents frequently fail by fixating on initial strategies without systematic exploration—precisely the failure mode this strategy addresses. Community-reported observations from the Hyperspace distributed run (McKinney 2026 ) suggest hardware-constrained diversity across 35 agents produced non-redundant discoveries; this is cited as qualitative corroboration only. Exploration–exploitation balance. Explicitly exploratory; the instruction mandates coverage breadth. Modification emphasis. Balanced coverage across all modification categories, with equal attention to architectural, optimiser, training dynamics, and initialisation changes. Modification emphasis. Balanced coverage across all modification categories, with equal attention to architectural, optimiser, training dynamics, and initialisation changes. Primary failure mode. Time fragmentation if the per-category cap is set too tightly for the session length. Conditions of applicability. Most appropriate for early-stage exploratory sessions, or when prior sessions have converged to a suspected local optimum. 5.5 Simplicity-Biased Strategy The reference autoresearch program.md (Karpathy 2026 ) includes an explicit simplicity criterion: modifications producing marginal metric improvements accompanied by significant code complexity are not retained; simplifications yielding equal or better performance are actively rewarded. The practical justification is transferability. GitHub Discussions (Karpathy 2026 , Discussion #43) document that approximately 20 autoresearch -discovered improvements transferred from depth-12 to depth-24 models; secondary reporting (Schmid 2026 ) indicates an 11% speedup at depth-24. These transfer claims require independent replication but are directionally consistent with the theoretical prediction that simpler modifications carry fewer context-specific assumptions. AlphaEvolve's discovery of the 48-scalar-multiplication matrix multiplication algorithm (Novikov et al. 2025 ) and FunSearch's cap-set discovery (Romera-Paredes et al. 2023 ) demonstrate that simplicity-biased evolutionary search can yield results of genuine scientific significance—both algorithms are simultaneously the highest-performing known and structurally the most parsimonious. Exploration–exploitation balance. Neutral; biases exploitation within the discovered region toward simpler solutions. Primary failure mode. Suppression of high-gain complex modifications. Mitigation: specify simplicity acts as a tiebreaker, not a veto, above an explicit threshold. Conditions of applicability. Whenever discovered optimisations are expected to transfer across model scales, architectures, or tasks. 5.6 Curriculum-Staged Strategy A Curriculum-Staged program divides the session into phases with distinct strategic objectives, using convergence-based conditions to trigger transitions. The theoretical motivation draws on curriculum learning (Bengio et al. 2009 ). AIRA (Toledo et al. 2025 ) provides direct quantitative support: pure exploitation is dominated by more exploratory policies in sessions exceeding approximately 40 experiments, and the performance gap widens with session length. This is directly analogous to the finding in MLAgentBench (Huang et al. 2024 ) that agents struggle with long-term planning—curriculum staging explicitly addresses this by structuring the long-horizon plan into phases with clear objectives and transition conditions. A three-phase structure is proposed as a calibration starting point: Phase 1 (coarse exploration, ≈ 30% of experiments)—wide category coverage, accepting improvements > 0.0005 val_bpb, transitioning when fewer than 20% of the last ten experiments improve; Phase 2 (targeted refinement, ≈ 50%)—focus on two to three most productive categories, threshold 0.0002; Phase 3 (robustness verification, ≈ 20%)—conservative threshold 0.0001. These thresholds are calibration starting points requiring empirical validation. AI Scientist-v2 (Yamada et al. 2025 ) independently instantiates this principle with progress-triggered phase transitions. Exploration–exploitation balance. Dynamically adaptive: E→X→conservative. Primary failure mode. Mis-timed phase transitions. Conditions of applicability. Sessions of 60 or more experiments. Table 3 Summary of the five instruction strategy families. E–E = exploration–exploitation balance; E = exploratory, X = exploitative. Families are not mutually exclusive Family E–E Balance Core Mechanism Primary Failure Mode Best Applied When Scope-Constrained Strongly X Linguistic scope restriction reduces search dimensionality Premature convergence without headroom Component known under-optimised Hypothesis-Directed Focused X Mechanistic rationales trigger chain-of-thought Anchoring bias Strong practitioner domain theory Diversity-Preserving E (mandated) Category-rotation prevents hyperparameter fixation Time fragmentation Early-stage; suspected local optimum Simplicity-Biased Neutral; biases within region Complexity penalty + metric tiebreaker; deletions valued Suppression of high-gain complex changes Transferability or interpretability required Curriculum-Staged Adaptive E→X→conservative Convergence-triggered phase transitions Mis-timed transitions Sessions ≥ 60 experiments 6 Empirical Grounding and Cross-System Validation 6.1 Autoresearch Session Analysis The two publicly documented autoresearch sessions (Karpathy 2026 ) provide directly interpretable evidence. The architecture makes program.md effects analytically visible: hardware, evaluation metric, and agent LLM are held constant within each session, and every outcome is recorded with a numerical val_bpb score. Before presenting the session comparison, the following methodological caveat is essential: Sessions A and B differ on two variables simultaneously. First, the instruction document content—Session B injected Session A's top discoveries as starting priors. Second, session length—Session A ran 89 experiments, Session B ran 126, a 42% difference. On a holistic per-experiment basis, Session B was marginally less efficient than Session A (0.0282/126 = 0.000224 per experiment versus 0.0206/89 = 0.000231 per experiment). The 37% total gain advantage of Session B is thus consistent with being explained by the additional 42% of experiments alone, if per-experiment efficiency were constant. This session-length confound prevents clean causal attribution of total gain differences to the instruction intervention. With this caveat stated, the session data contains a pattern that session length alone cannot account for: the qualitative composition of what was discovered. Session A (Discussion #32, 89 experiments) improved val_bpb from 0.9979 to 0.9773 without cross-session priors; its primary discoveries were batch halving, depth 9, SSSSL window pattern, and RoPE 200K. Session B (Discussion #43, 126 experiments) achieved total gain 0.0282; its primary discoveries were weight decay on embedding and value projection parameters, and transformer initialisation scale. Session A had 89 experiments to discover weight decay strategies and did not; Session B found them within the first 30 experiments after prior injection. This qualitative compositional difference is not explained by session length and is consistent with the Curriculum-Staged family's cross-session memory mechanism. Table 4 summarises metrics. Table 4 Comparative metrics for Sessions A and B (Karpathy 2026 ). The 37% total gain difference is length-confounded and must not be interpreted as a causal estimate of the instruction intervention effect Metric Session A Session B Interpretation Total experiments 89 126 + 42% more in B — confound Starting val_bpb 0.9979 0.9979 Identical baseline Total val_bpb gain 0.0206 0.0282 + 37% in B; length-confounded (see text) Holistic per-experiment efficiency 0.000231 0.000224 B marginally less — length alone could explain total gap Experiments to first 0.010 gain ~ 15 ~ 5 67% reduction — exploratory phase compressed Late-phase efficiency (exp 16+; post-hoc window) ~ 0.000089 ~ 0.00021 2.36× (post-hoc segmentation; no uncertainty estimate) Primary discoveries Batch halving; depth 9; SSSSL; RoPE 200K WD on embeddings; init scale 0.68× Qualitatively different — not explained by length alone 6.2 Evidence from the Agentic AutoML Literature The peer-reviewed literature provides controlled experimental support for the taxonomy's predictions. AgentHPO (Liu et al. 2024a ) directly validates the Hypothesis-Directed family under controlled conditions: across twelve benchmark HPO tasks, agents with mechanistic rationales required significantly fewer trials than unguided baseline agents. Specifically, the paper reports that hypothesis-directed agents reached competitive performance in roughly half the number of evaluations compared to the baseline in the median case across the twelve tasks, with improvements observed on all tasks. The mechanism—chain-of-thought priming (Wei et al. 2022 )—operates at every agent call, priming the model to reason about expected experimental outcomes before executing modifications. R&D-Agent (Yang et al. 2025 ) independently validates this at system level: its 35.1% MLE-bench medal rate compared to AIDE's 16.9% (Chan et al. 2024 ) provides controlled evidence that structured hypothesis separation (Researcher vs. Developer role) roughly doubles system performance. AIRA (Toledo et al. 2025 ) directly validates the Curriculum-Staged family. Their systematic comparison of Greedy, MCTS, and Evolutionary policies across MLE-bench tasks demonstrates that pure exploitation is dominated by more exploratory policies in sessions exceeding approximately 40 experiments, and the performance gap widens with session length. AIRA also documents 9–13 percentage-point performance gaps from validation-set overfitting in long sessions. MLAgentBench (Huang et al. 2024 ) provides independent corroboration of the long-term planning challenge: success rate drops from 100% on short, well-defined tasks to 0–25% on open-ended Kaggle challenges, consistent with the prediction that undifferentiated uniform strategies fail in extended sessions. AlphaEvolve (Novikov et al. 2025 ) and FunSearch (Romera-Paredes et al. 2023 ) jointly validate the Simplicity-Biased family. The Diversity-Preserving family is supported by AIRA's finding that evolutionary policies outperform greedy in extended sessions and by population-based training theory (Jaderberg et al. 2017 ). 6.3 Cross-System Taxonomy Validation Table 5 maps each family to supporting systems, distinguishing explicit instantiation from implicit corroboration. Table 5 Cross-system validation of the five taxonomy families. Explicit = family explicitly instantiated in system's design; Implicit = effects observable but not named as such Family Supporting Systems Nature of Evidence Scope-Constrained AIRS-Bench; AIDE; R&D-Agent; MLAgentBench Explicit in AIRS-Bench task definitions and R&D-Agent task briefs. Implicit in AIDE and MLAgentBench task specifications. autoresearch data: tenfold efficiency decline consistent with absence of scope constraint. Hypothesis-Directed AgentHPO; R&D-Agent; AI-Researcher; DS-Agent Quantified and controlled in AgentHPO: structured hypothesis provision reduces trials across 12 HPO tasks (Explicit). R&D-Agent: researcher–developer separation credited as key driver (35.1% vs. 16.9% medal rate). AI-Researcher Idea Generator module (Explicit). DS-Agent: case-based reasoning (equivalent mechanism). Diversity-Preserving AlphaEvolve; FunSearch; AIRA (evolutionary); MLAgentBench Explicit diversity mechanisms in AlphaEvolve (dual-model) and FunSearch (island populations). AIRA: evolutionary policy outperforms greedy in extended sessions (controlled). MLAgentBench: agent strategy fixation identified as primary failure mode. Hyperspace observations: qualitative corroboration only (McKinney 2026 ). Simplicity-Biased autoresearch; AlphaEvolve; FunSearch; AI Scientist Explicit in reference program.md simplicity criterion. Transfer evidence from GitHub Discussions (Karpathy 2026 ) and Schmid ( 2026 ) — requires independent replication. AlphaEvolve: 48-scalar matrix multiplication (highest-performing + structurally simplest). FunSearch: cap-set discovery. Curriculum-Staged AIRA; AI Scientist-v2; autoresearch; Agent Lab; MLAgentBench Quantified and controlled in AIRA: greedy dominated by exploratory policies > 40 experiments; gap widens with length. MLAgentBench: agent success drops on long-horizon tasks (0–25%), validating long-session planning challenge. AI Scientist-v2: phase-structured pipeline. autoresearch: qualitative territory change (length-confounded for total gain). 6.4 Quantitative Performance Landscape To contextualise the practical significance of instruction design, Table 6 summarises quantitative performance benchmarks across systems for which standardised results are available. This table illustrates the performance gap across system designs—a gap that the present paper argues is substantially determined by the quality of instruction documents, particularly the Hypothesis-Directed and Scope-Constrained design choices that distinguish high-performing systems. Table 6 Quantitative performance overview of systems with available MLE-bench or standardised benchmark results. All MLE-bench figures are 'any medal rate' (%). MLAgentBench figures are task success rate (%). Results reflect best published configuration at time of survey (March 2026). Sources: Chan et al. ( 2024 ), Toledo et al. ( 2025 ), Yang et al. ( 2025 ), Huang et al. ( 2024 ) System Benchmark Best Published Result Key Instruction Feature Notes R&D-Agent (Yang et al. 2025 ) MLE-bench 35.1% any medal Explicit hypothesis–implementation separation SOTA open-source at survey cutoff; dual-agent design AIRA — Evolutionary (Toledo et al. 2025 ) MLE-bench >AIDE greedy in sessions > 40 exp Diversity-preserving evolutionary search policy Controlled policy comparison; gap widens with length AIDE + o1-preview (Jiang et al. 2025 ) MLE-bench 16.9% any medal Greedy tree search; no explicit hypothesis seeding Baseline for comparison; doubles to 34.1% at pass@8 AgentHPO (Liu et al. 2024a ) HPO (12 tasks) ~ 50% fewer trials vs. baseline Hypothesis-directed with mechanistic rationales Controlled study; improvements across all 12 tasks MLAgentBench—Claude Opus (Huang et al. 2024 ) MLAgentBench 37.5% task success (best agent) ReAct-style; no structured curriculum 0% on recent Kaggle tasks; 100% on simple tasks AlphaEvolve (Novikov et al. 2025 ) Algorithm discovery 48-scalar matmul; 0.7% DC efficiency Diversity-preserving dual-model; simplicity bias Scientific significance: new mathematical result The quantitative landscape in Table 6 reveals a clear pattern: systems with explicit hypothesis-directed and diversity-preserving instruction mechanisms outperform those relying on undifferentiated greedy search. The 35.1% vs. 16.9% medal rate gap between R&D-Agent and AIDE on identical benchmarks—with the key architectural difference being the Hypothesis-Directed design of R&D-Agent's Researcher component—provides the clearest available controlled evidence that instruction strategy design has measurable, substantial impact on system performance. 6.5 Validation-Set Overfit Risk Any iterative optimisation system re-querying a fixed validation set risks progressive overfit. AIRA (Toledo et al. 2025 ) quantifies this: selecting the final configuration by held-out test-set score rather than validation score improved performance by 9–13 percentage points across MLE-bench tasks. Instruction design can mitigate this risk: the Curriculum-Staged Phase 3 conservative acceptance threshold limits configurations accumulating marginal overfit gains; the Simplicity-Biased strategy's preference for architectural modifications reduces the surface area available for validation-set exploitation. 6.6 Answer to RQ3 Multi-source evidence corroborates all five taxonomy families. The strongest controlled evidence comes from AgentHPO for the Hypothesis-Directed family and AIRA for the Curriculum-Staged family. The quantitative performance landscape in Table 6 demonstrates that systems implementing Hypothesis-Directed and Diversity-Preserving instruction mechanisms achieve substantially higher benchmark performance than systems without these features. The autoresearch session comparison provides suggestive but non-causal evidence; the session-length confound prevents attribution of the 37% total gain difference to instruction design alone. 7 Practitioner Guidelines Five guidelines are synthesised for practitioners authoring instruction documents. All numerical thresholds are calibration starting points requiring empirical validation in specific deployment contexts. Guideline 1 — Inject cross-session priors before any follow-on run. Update program.md to include all discoveries from prior sessions as established starting configurations before beginning a second or subsequent session. This is the highest-leverage intervention suggested by the available evidence: it compresses the exploratory phase (approximately 67% reduction in experiments-to-first-categorical-gain in the autoresearch sessions), redirects session attention toward qualitatively new territory, and is consistent with the Curriculum-Staged strategy's cross-session memory mechanism. The 37% total gain advantage attributed to this intervention is length-confounded (Sect. 6.1 ) and should not be interpreted as a controlled causal estimate. Guideline 2 — Stage the instruction for sessions of 60 or more experiments. Implement a two- or three-phase curriculum with convergence-rate transitions. Suggested calibration starting points: fewer than 20% improvement rate in the last ten experiments as the Phase 1-to-Phase 2 transition condition; rising acceptance thresholds in Phase 3. AIRA's controlled comparison (Toledo et al. 2025 ) provides the motivation: greedy is dominated by exploratory policies in extended sessions. For sessions of 40 experiments or fewer, curriculum management overhead is unlikely to be worthwhile. Guideline 3 — Pair every hypothesis with a mechanistic rationale. Rather than listing modifications to attempt, explain why each is expected to improve performance given the session's specific compute constraints. AgentHPO (Liu et al. 2024a ) demonstrates roughly 50% reduction in trials across 12 tasks; R&D-Agent (Yang et al. 2025 ) demonstrates a more than doubling of MLE-bench medal rate compared to unstructured approaches (35.1% vs. 16.9%). As the session progresses, authorise the agent to generate its own rationale-equipped hypotheses, preventing anchoring bias. Guideline 4 — Encode a category-rotation directive for exploratory sessions. Specify a cap of 8–10 consecutive experiments within any single modification category for a 100-experiment session, with mandatory rotation across at least five distinct categories. This starting point (cap ≈ 0.08–0.10 × total planned experiments) is motivated by the autoresearch session data showing tenfold efficiency decline without a diversity directive and by MLAgentBench evidence (Huang et al. 2024 ) of strategy fixation as a primary failure mode. Guideline 5 — Calibrate the simplicity criterion explicitly. Specify that simplicity acts as a tiebreaker—not a veto—for metric-equivalent proposals. Suggested starting threshold: improvements exceeding 0.002 val_bpb are accepted regardless of added complexity; improvements below this threshold favour the simpler option. This threshold is illustrative and should be adjusted based on typical early-session gain magnitudes. 8 Limitations and Future Research Directions 8.1 Limitations Five categories of limitation apply. First, the primary observational evidence involves only two autoresearch sessions with no controlled random seed matching. The session-length confound (89 vs. 126 experiments) prevents clean causal attribution. All session comparison figures should be treated as descriptive statistics from non-randomised observations. Second, the taxonomy was developed inductively by a single author from a specific framework (LLM pretraining, 5-minute runs, single GPU). Single-author taxonomy development lacks the inter-rater reliability of multi-coder protocols. The saturation criterion and cross-system validation in Sect. 6.3 partially address this. Third, agent behaviour depends substantially on the LLM backbone. The optimal instruction strategy for a reasoning-capable model (e.g., capable of strong chain-of-thought) likely differs from what is optimal for weaker models. The performance sensitivity identified by MLAgentBench (Huang et al. 2024 )—ranging from 100% success on simple tasks to 0% on open-ended challenges—illustrates that task characteristics also moderate instruction strategy effectiveness in ways our taxonomy does not fully capture. Fourth, some empirical claims rest on grey literature: the Simplicity-Biased transferability claim relies on GitHub Discussions and a secondary blog post (Schmid 2026 ); the Diversity-Preserving Hyperspace evidence relies on media coverage (McKinney 2026 ). These should be treated as directionally indicative until independently replicated. Fifth, the landscape is rapidly evolving. New systems have emerged since the survey cutoff, and the autoresearch repository continues receiving community contributions. The taxonomy is a snapshot of a field in motion. 8.2 Future Research Directions Controlled experimental comparison of program variants. The highest-priority next step is a controlled comparison of instruction strategies with matched random seeds, identical hardware, and identical session lengths. Even a 2×2 factorial design (Diversity-Preserving vs. Scope-Constrained × with vs. without cross-session priors) across ten matched session pairs would provide the first controlled causal evidence for the predictions this taxonomy makes. Automated program optimisation. If instruction documents are the primary practitioner-facing control, optimising them is itself an optimisation problem. The prompt engineering literature provides gradient-free methods for automatic single-turn prompt optimisation (Li et al. 2025 ); LLaMEA-style LLM evolutionary algorithms (van Stein and Bäck 2024 ) and in-the-loop HPO for LLM-based heuristic design (van Stein et al. 2026 ) provide methodological foundations that could be extended to research program optimisation. Standardised instruction quality evaluation protocols. The field lacks a benchmark for instruction document quality comparable to MLE-bench (Chan et al. 2024 ) for agent architecture quality. Developing a standardised evaluation protocol—specifying matched session conditions, control programs, and quality metrics—would enable systematic comparison of instruction strategies across the research community. Multi-agent coordination through complementary program assignments. Assigning complementary instruction strategies to coordinated agents would extend the Diversity-Preserving strategy from single-agent instruction design to multi-agent program design, building on insights from the Hyperspace distributed run (McKinney 2026 ). LLM-specific program design optimisation. The optimal program design likely varies with backbone LLM characteristics. A systematic study of strategy–LLM interaction effects would allow the guidelines in Sect. 7 to be refined with model-specific calibrations. 9 Conclusions This paper has addressed a gap at the intersection of autonomous machine learning experimentation and LLM instruction engineering. In autonomous experimentation systems—from the 16.9% medal-rate AIDE to the 35.1% medal-rate R&D-Agent to the 37.5% task-success MLAgentBench baseline—the practitioner's primary lever is the instruction document that directs the agent across an extended session of dozens or hundreds of consecutive decisions. Yet this document has received almost no analytical attention as an engineering artefact. Four contributions have been made. First, a structured survey of sixteen agentic AutoML and autonomous research systems—positioned within a new related work section that situates the paper against existing surveys—establishing that the instruction document is a universal practitioner-facing control mechanism, exposing seven design dimensions. Second, a five-family taxonomy grounded in theory from the AutoML, evolutionary computation (Bäck et al. 2023 ; van Stein and Bäck 2024 ), prompt engineering, and curriculum learning literatures, with each family characterised along four analytical dimensions and cross-validated against at least two independent systems. Third, multi-source empirical grounding that links the taxonomy's predictions to controlled experimental evidence: the 35.1% vs. 16.9% medal rate gap between R&D-Agent and AIDE—attributable to hypothesis-directed instruction design—provides the clearest available controlled evidence that instruction strategy has measurable, substantial impact on system performance. Fourth, five practitioner guidelines with explicitly labelled calibration thresholds, validated against all sixteen systems. The key finding is not merely that instruction document quality matters—it is that principled, theoretically-grounded instruction strategy design explains a substantial fraction of the observed performance variation across current state-of-the-art systems. As the research community moves from individual system development toward benchmarked comparison on standardised tasks, understanding and engineering the instruction layer will become as important as engineering the search algorithm or the agent architecture. The taxonomy, performance landscape, and research agenda presented here provide the foundation for that work. Declarations Funding The author did not receive support from any organisation for the submitted work. No funds, grants, or other support was received. Competing Interests The author has no relevant financial or non-financial interests to disclose. Author Contributions Praneeth Kodumagulla: Conceptualization, Formal analysis, Investigation, Methodology, Writing—original draft, Writing—review and editing. The author has read and agreed to the submitted version of the manuscript. Data Availability No new primary datasets were created in this study. The autoresearch session logs analysed in Sect. 6.1 are publicly available as GitHub Discussions #32 and #43 at https://github.com/karpathy/autoresearch/discussions (accessed 26 March 2026). The autoresearch codebase is available at https://github.com/karpathy/autoresearch under the MIT License. All other empirical evidence is drawn from publicly available peer-reviewed papers and preprints as cited. Ethics Approval Not applicable. This study involves no human participants, no animal subjects, and no personal data. References Bäck THW, Kononova AV, van Stein B, Wang H, Antonov KA, Kalkreuth RT, de Nobel JP, Vermetten D, de Winter R, Ye F (2023) Evolutionary algorithms for parameter optimization—thirty years later. Evol Comput 31(2):81–122. https://doi.org/10.1162/evco_a_00325 Baratchi M, Wang C, Limmer S, van Rijn JN, Hoos H, Bäck T, Olhofer M (2024) A literature review on automated machine learning. Artif Intell Rev. https://doi.org/10.1007/s10462-024-10726-1 Bengio Y, Louradour J, Collobert R, Weston J (2009) Curriculum learning. In: Proceedings of the 26th annual international conference on machine learning (ICML 2009). ACM, New York, pp 41–48. https://doi.org/10.1145/1553374.1553380 Chan JS, Chowdhury N, Jaffe O, Aung J, Sherburn D, Mays E, Starace G, Liu K, Maksin L, Patwardhan T, Weng L, Mądry A (2024) MLE-bench: evaluating machine learning agents on machine learning engineering. arXiv:2410.07095 Chi Y, Lin Y, Hong S, Pan D, Fei Y, Mei G, Liu B, Pang T, Kwok J, Zhang C (2024) SELA: tree-search enhanced LLM agents for automated machine learning. arXiv:2410.17238 Gardner J, Dutta A, Bölöni L (2026) From language to action: a review of large language models as autonomous agents and tool users. Artif Intell Rev. https://doi.org/10.1007/s10462-025-11471-9 Guo J, Li Z, Ma D, Zheng T, Zang Q, Liu M, Niu X, Wang Y, Yang J, Liu J (2024) DS-Agent: automated data science by empowering large language models with case-based reasoning. arXiv:2402.17453 Huang Q, Vora J, Liang P, Leskovec J (2024) MLAgentBench: evaluating language agents on machine learning experimentation. In: Proceedings of the 41st international conference on machine learning (ICML 2024). PMLR, pp 20265–20289. arXiv:2310.03302 Jaderberg M, Dalibard V, Osindero S, Czarnecki WM, Donahue J, Razavi A, Vinyals O, Green T, Dunning I, Simonyan K (2017) Population based training of neural networks. arXiv:1711.09846 [preprint] Jiang Z, Schmidt D, Srikanth D, Xu D, Kaplan I, Jacenko D, Wu Y (2025) AIDE: AI-driven exploration in the space of code. arXiv:2502.13138 Karpathy A (2026) autoresearch: AI agents running research on single-GPU nanochat training automatically. GitHub repository. https://github.com/karpathy/autoresearch . Accessed 26 Mar 2026 Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007-001. Keele University, Keele Li W, Li Z, Li Y (2025) A survey of automatic prompt engineering: an optimization perspective. arXiv:2502.11560 Li Z, Zang Q, Ma D, Guo J, Zheng T, Liu M, Niu X, Wang Y, Yang J, Liu J, Zhong W, Zhou W, Huang W, Zhang G (2024b) AutoKaggle: a multi-agent framework for autonomous data science competitions. arXiv:2410.20424 Liu S, Chen J, Ye S, Li Q, Li J, Yu J, Liu Y, Li H, Sun Y, Ying R (2024a) AgentHPO: large language model agent for hyper-parameter optimization. arXiv:2402.01881 Lu C, Lu C, Lange RT, Foerster J, Clune J, Ha D (2024) The AI Scientist: towards fully automated open-ended scientific discovery. arXiv:2408.06292 Lupidi A, Gauri B, Foster TS, Al Omari B, Magka D, Pepe A, Audran-Reiss A et al (2026) AIRS-Bench: a suite of tasks for frontier AI research science agents. arXiv:2602.06855 McKinney E (2026) Andrej Karpathy's new open source 'autoresearch' lets you run hundreds of AI experiments a night. VentureBeat, 10 Mar 2026. https://venturebeat.com/technology/andrej-karpathys-new-open-source-autoresearch-lets-you-run-hundreds-of-ai . Accessed 26 Mar 2026 [media report; not peer-reviewed] Nathani D, Madaan L, Roberts N, Bashlykov N, Menon A, Moens V, Budhiraja A, Magka D, Vorotilov V, Chaurasia G, Hupkes D, Cabral RS, Shavrina T, Foerster J, Bachrach Y, Wang WY, Raileanu R (2025) MLGym: a new framework and benchmark for advancing AI research agents. arXiv:2502.14499 Novikov A, Vu N, Eisenberger M, Dupont E, Huang P-S, Wagner AZ, Shirobokov S, Kozlovskii B, Ruiz FJR, Mehrabian A (2025) AlphaEvolve: a coding agent for scientific and algorithmic discovery. arXiv:2506.13131 Petersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic mapping studies in software engineering. In: Proceedings of the 12th international conference on evaluation and assessment in software engineering (EASE 2008). BCS Learning & Development, Swindon, pp 68–77 Plaat A et al (2025) Agentic AI: a comprehensive survey of architectures, applications, and future directions. Artif Intell Rev. https://doi.org/10.1007/s10462-025-11422-4 Renduchintala HSVNSK, Panwar P, Singh P, Khandelwal A (2024) POSIX: a prompt sensitivity index for large language models. arXiv:2410.02185 Romera-Paredes B, Barekatain M, Novikov A, Balog M, Kumar MP, Dupont E, Ruiz FJR, Ellenberg JS, Wang P, Fawzi O (2023) Mathematical discoveries from program search with large language models. Nature 624:545–552. https://doi.org/10.1038/s41586-023-06924-6 Sclar M, Choi Y, Tsvetkov Y, Suhr A (2024) Quantifying language models' sensitivity to spurious features in prompt design or: how I learned to start worrying about prompt formatting. In: Proceedings of ICLR 2024. arXiv:2310.11324 Schmid P (2026) How autoresearch will change small language models adoption. philschmid.de. https://www.philschmid.de/autoresearch . Accessed 26 Mar 2026 [secondary report; claims require independent replication] Schmidgall S, Su Y, Wang Z, Sun X, Wu J, Yu X, Liu J, Liu Z, Barsoum E (2025) Agent Laboratory: using LLM agents as research assistants. arXiv:2501.04227 Snoek J, Larochelle H, Adams RP (2012) Practical Bayesian optimization of machine learning algorithms. Advances in neural information processing systems, vol 25. Curran Associates, Red Hook, pp 2951–2959 Tang J, Xia L, Li Z, Huang C (2025) AI-Researcher: autonomous scientific innovation. In: Advances in neural information processing systems. Curran Associates, Red Hook. arXiv:2505.18705 Toledo E, Hambardzumyan K, Josifoski M, Hazra R, Baldwin N, Audran-Reiss A, Kuchnik M, Magka D, Jiang M, Lupidi AM et al (2025) AI research agents for machine learning: search, exploration, and generalisation in MLE-bench. arXiv:2507.02554 (v2, November 2025) van Stein N, Bäck THW (2024) LLaMEA: a large language model evolutionary algorithm for automatically generating metaheuristics. IEEE Trans Evol Comput. https://doi.org/10.1109/TEVC.2024.3511703 van Stein N, Vermetten DL, Bäck THW (2026) In-the-loop hyper-parameter optimization for LLM-based automated design of heuristics. ACM Trans Evol Learn Optim. https://doi.org/10.1145/3729220 Wang L, Ma C, Feng X, Zhang Z, Yang H, Zhang J, Chen Z, Tang J, Chen X, Lin Y, Zhao WX, Wei Z, Wen JR (2024) A survey on large language model based autonomous agents. Front Comput Sci 18(6):186345. https://doi.org/10.1007/s11704-024-40231-1 Wang Z, Wang K, Wang Q, Zhang P, Li L, Yang Z, Jin X, Yu K, Nguyen MN, Liu L, Gottlieb E, Lu Y, Cho K, Wu J, Fei-Fei L, Wang L, Choi Y, Li M (2025) RAGEN: understanding self-evolution in LLM agents via multi-turn reinforcement learning. arXiv:2504.20073 Wei J, Wang X, Schuurmans D, Bosma M, Xia F, Chi E, Le QV, Zhou D (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, vol 35. Curran Associates, Red Hook, pp 24824–24837 Wistuba M, Schilling N, Schmidt-Thieme L (2015) Hyperparameter search space pruning—a new component for sequential model-based hyperparameter optimization. In: Proceedings of ECML PKDD 2015. Springer, Berlin, pp 104–119 Yamada Y, Lange RT, Lu C, Hu S, Lu C, Foerster J, Clune J, Ha D (2025) The AI Scientist-v2: workshop-level automated scientific discovery via agentic tree search. arXiv:2504.08066 Yang X, Yang X, Fang S, Xian B, Li Y, Wang J, Xu M, Pan H, Hong X, Liu W, Shen Y, Chen W, Bian J et al (2025) R&D-Agent: an LLM-agent framework towards autonomous data science. arXiv:2505.14738 (v2, October 2025) Zheng Y, Zhang L, Li K, Wang R, Li W, Zhang T, Zhang Q et al (2026) A survey on large language models driven meta-optimizers for automated intelligent optimization. Artif Intell Rev 59:72. https://doi.org/10.1007/s10462-025-11470-w Zhuo J, Zhang S, Fang X, Duan H, Lin D, Chen K (2024) ProSA: assessing and understanding the prompt sensitivity of LLMs. Findings of ACL: EMNLP 2024. ACL, Miami, pp 1950–1976 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Reviews received at journal 10 May, 2026 Reviews received at journal 06 May, 2026 Reviews received at journal 04 May, 2026 Reviews received at journal 30 Apr, 2026 Reviewers agreed at journal 22 Apr, 2026 Reviewers agreed at journal 21 Apr, 2026 Reviewers agreed at journal 20 Apr, 2026 Reviewers agreed at journal 19 Apr, 2026 Reviewers agreed at journal 19 Apr, 2026 Reviewers invited by journal 19 Apr, 2026 Editor assigned by journal 06 Apr, 2026 Submission checks completed at journal 02 Apr, 2026 First submitted to journal 01 Apr, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9286871","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":615658598,"identity":"a8dc2c3a-9b67-4eba-aa88-45e280e9c297","order_by":0,"name":"Praneeth Kodumagulla","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA60lEQVRIiWNgGAWjYBCDBAMQ+QGI2diJ02EA1sI4A6SFmRQtzDwgNiEt5tKHnz34UPEnz5y99+Fjm1/b5PmYGRg/fMzBrcWyL83ccMYZg2LLnuPGxrl9tw3bmBmYJWduw+OiMwxm0rxtBokbbqSxSef23GYEamFj5sWrhf2b9N9/QC33n7H/tuy5bU+EFh4zacYGkC1swLD6cTuRoBbLHp4yyZ5jxokbzqQxS/Y23E5uY2ZsxusXcx72bRI/auQSNxw/xvjhx5/btvPbmw9++IjPYSg8xjYw2YBbPYYWhj94FY+CUTAKRsEIBQAUAk8fKFrgjgAAAABJRU5ErkJggg==","orcid":"","institution":"Independent Researcher","correspondingAuthor":true,"prefix":"","firstName":"Praneeth","middleName":"","lastName":"Kodumagulla","suffix":""}],"badges":[],"createdAt":"2026-04-01 05:24:42","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9286871/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9286871/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[],"financialInterests":"No competing interests reported.","formattedTitle":"Instruction Strategy Design for Autonomous Machine Learning Experimentation Systems: A Taxonomy, Cross-System Analysis, and Evidence-Based Practitioner Framework","fulltext":[{"header":"1 Introduction","content":"\u003cp\u003eThe practice of machine learning (ML) research has long been bounded by a fundamental throughput constraint: the human researcher. Each experimental cycle\u0026mdash;formulate a hypothesis, write and execute code, interpret metrics, decide whether to proceed\u0026mdash;requires skilled judgment at every step, with wall-clock times measured in hours. Even in well-resourced settings, a single researcher executes only a handful of substantive experiments per day, making this the binding limit on how quickly the community can explore model configurations, architectures, and training strategies.\u003c/p\u003e \u003cp\u003eThe emergence of LLMs capable of reading, writing, and reasoning about code has created a qualitatively different possibility. By delegating the operational layer to an autonomous agent, the human researcher's role shifts from direct experimentation to strategic direction-setting. An overnight session on a single GPU can execute 80\u0026ndash;100 experiments; distributed multi-agent deployments can cover hundreds more, representing throughput improvements of roughly two orders of magnitude over manual iteration (Karpathy \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2026\u003c/span\u003e; Toledo et al. \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). The significance of this shift is reflected in the performance of current systems on the MLE-bench benchmark (Chan et al. \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2024\u003c/span\u003e)\u0026mdash;75 real-world Kaggle ML engineering competitions: AIDE with o1-preview achieves a medal rate of 16.9%, R\u0026amp;D-Agent achieves 35.1%, and the best current open-source system achieves a score that would have been inconceivable without autonomous experimentation. Yet despite this progress, the instruction documents that direct these systems remain entirely unanalysed as an engineering artefact.\u003c/p\u003e \u003cp\u003eThis paradigm sits at the intersection of two active research frontiers. The first is LLM-driven automated algorithm design and meta-optimisation\u0026mdash;including LLaMEA, which uses GPT to generate and iteratively evolve complete metaheuristic algorithm code (van Stein and B\u0026auml;ck \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), and LLM-driven meta-optimisers for automated intelligent optimisation more broadly (Zheng et al. \u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e2026\u003c/span\u003e). The second is the emergence of production-scale autonomous ML research systems: \u003cem\u003eautoresearch\u003c/em\u003e (Karpathy \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2026\u003c/span\u003e), which accumulated more than 50,000 GitHub stars within three weeks of release; AIDE (Jiang et al. \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e); AIRA (Toledo et al. \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2025\u003c/span\u003e); R\u0026amp;D-Agent (Yang et al. \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e2025\u003c/span\u003e); AgentHPO (Liu et al. \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2024a\u003c/span\u003e); AlphaEvolve (Novikov et al. \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2025\u003c/span\u003e); and MLAgentBench (Huang et al. \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e)\u0026mdash;which established the foundational benchmark for evaluating LLM agents on ML experimentation tasks (13 tasks, best agent 37.5% success rate with Claude 3 Opus), predating and motivating the current generation of systems.\u003c/p\u003e \u003cp\u003eAcross all these systems, one structural feature is universal and entirely unstudied: the natural-language instruction document that is the sole artefact under deliberate human control once a session begins. In \u003cem\u003eautoresearch\u003c/em\u003e this is program.md; in AIDE it is the task specification; in AIRS-Bench (Lupidi et al. \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) it is project_description.md; in MLAgentBench (Huang et al. \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) it is the task description. This document determines what the agent considers a valid modification, how it balances exploration and exploitation, what prior knowledge it carries from previous sessions, and how it behaves when progress stalls. Yet the design of this document has received almost no analytical attention in the literature. The contrast with related work in automated algorithm configuration (B\u0026auml;ck et al. \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) and LLM-based heuristic design (van Stein and B\u0026auml;ck \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; van Stein et al. \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) is striking: those domains have rich theoretical frameworks for design-space navigation; the analogous problem for autonomous ML research programs does not.\u003c/p\u003e \u003cp\u003eThe prompt engineering literature establishes that LLM behaviour is strongly sensitive to instruction wording even in single-turn inference tasks\u0026mdash;with documented performance differences of up to 76 accuracy points across semantically equivalent prompt formats (Sclar et al. \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). In an autonomous experimentation session we hypothesise that these effects compound: each agent decision is conditioned on prior decisions and their outcomes, so an instruction that biases the agent toward unproductive territory in experiment 10 also shapes the context for experiments 11 through 100. This compounding hypothesis is supported by AIRA's finding (Toledo et al. \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) that search-policy performance differences widen with session length\u0026mdash;a prediction directly analogous to the session-length sensitivity observed in hyperparameter optimisation (Snoek et al. \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2012\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThis paper addresses the gap through a structured survey and taxonomy. We survey sixteen agentic AutoML and autonomous research systems, extract the common role and design dimensions of their instruction documents, develop a five-family taxonomy grounded in theory and empirical evidence, validate the taxonomy across systems, and synthesise practitioner guidelines.\u003c/p\u003e \u003cp\u003e \u003cb\u003eContributions.\u003c/b\u003e Four specific contributions are made:\u003c/p\u003e \u003cp\u003e(1) A structured cross-system survey of sixteen agentic AutoML and autonomous research systems, identifying the instruction document as a universal practitioner-facing control mechanism and cataloguing seven design dimensions.\u003c/p\u003e \u003cp\u003e(2) A five-family taxonomy of instruction strategies grounded in theory from the AutoML, evolutionary computation, prompt engineering, and curriculum learning literatures, with each family characterised along four analytical dimensions.\u003c/p\u003e \u003cp\u003e(3) Multi-source empirical grounding drawn from publicly documented session logs, quantitative benchmarks from AIRA and AgentHPO under independently controlled conditions, and community-reported distributed observations; cross-validation of each family against at least two independent systems.\u003c/p\u003e \u003cp\u003e(4) Five practitioner guidelines with explicitly labelled calibration thresholds, a validation matrix, and a structured research agenda for future controlled study.\u003c/p\u003e \u003cp\u003eThe paper is organised as follows. Section \u003cspan refid=\"Sec2\" class=\"InternalRef\"\u003e2\u003c/span\u003e reviews related work and positions the paper with respect to existing surveys. Section \u003cspan refid=\"Sec6\" class=\"InternalRef\"\u003e3\u003c/span\u003e frames three research questions. Section \u003cspan refid=\"Sec7\" class=\"InternalRef\"\u003e4\u003c/span\u003e surveys the landscape. Section \u003cspan refid=\"Sec12\" class=\"InternalRef\"\u003e5\u003c/span\u003e develops the taxonomy. Section \u003cspan refid=\"Sec19\" class=\"InternalRef\"\u003e6\u003c/span\u003e provides empirical grounding. Section \u003cspan refid=\"Sec26\" class=\"InternalRef\"\u003e7\u003c/span\u003e derives practitioner guidelines. Section \u003cspan refid=\"Sec27\" class=\"InternalRef\"\u003e8\u003c/span\u003e discusses limitations and future directions. Section \u003cspan refid=\"Sec30\" class=\"InternalRef\"\u003e9\u003c/span\u003e concludes.\u003c/p\u003e"},{"header":"2 Related Work","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Surveys of Agentic AutoML and Autonomous Research Systems\u003c/h2\u003e \u003cp\u003eSeveral survey papers have addressed the broader landscape of LLM-based autonomous agents in scientific research and machine learning engineering. Wang et al. (\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) provide a comprehensive survey of LLM-based autonomous agents, establishing a unified framework covering memory, planning, action, and tool use\u0026mdash;but their survey addresses general-purpose agents rather than the specific class of systems that autonomously modify and evaluate code within a fixed evaluation loop. Gardner et al. (\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) review LLMs as autonomous agents and tool users with a focus on the role of tools and feedback loops in agentic behaviour; instruction document design falls outside their scope. The Agentic AI survey (Plaat et al. \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) provides a taxonomy of agentic architectures across symbolic and neural paradigms but does not address the design of the practitioner-facing control artefact. The AutoML literature review (Baratchi et al. \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) surveys classical AutoML methods but predates the LLM-driven autonomous experimentation paradigm introduced by AIDE, AIRA, and \u003cem\u003eautoresearch.\u003c/em\u003e The present paper occupies the gap between these surveys: it addresses the specific class of LLM-driven autonomous ML experimentation systems and, uniquely, focuses on the practitioner-facing instruction document rather than the agent architecture.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Prompt Engineering and Instruction Quality\u003c/h2\u003e \u003cp\u003eThe prompt engineering literature establishes sensitivity of LLM outputs to instruction wording (Sclar et al. \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Renduchintala et al. \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Zhuo et al. \u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) and provides gradient-free methods for single-turn prompt optimisation (Li et al. \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). However, this literature addresses single-turn inference tasks\u0026mdash;where each prompt is evaluated independently\u0026mdash;whereas research program design addresses multi-turn, state-accumulating experimentation sessions where each agent decision conditions all subsequent ones. LLaMEA (van Stein and B\u0026auml;ck \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) and the in-the-loop HPO work (van Stein et al. \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) address LLM-driven algorithm design in the evolutionary computation tradition; their focus is on the algorithm text rather than the practitioner-facing program specification. The present paper bridges these communities: it applies the analytical rigour of prompt sensitivity research to the multi-turn, session-level instruction design problem in autonomous ML research.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Benchmarks for Autonomous ML Experimentation\u003c/h2\u003e \u003cp\u003eMLAgentBench (Huang et al. \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) was the first systematic benchmark for evaluating LLM agents on ML experimentation, introducing 13 tasks and establishing that the best agent (Claude 3 Opus) achieves a 37.5% average task success rate but struggles with long-term planning. MLE-bench (Chan et al. \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) extended this to 75 real-world Kaggle competitions, finding that the best agent setup (AIDE with o1-preview) achieves a 16.9% medal rate at pass@1. Both benchmarks treat the instruction document as a fixed given rather than as a variable to be optimised. AIRS-Bench (Lupidi et al. \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) provides a 20-task benchmark where project_description.md files are the primary control variable, making it the closest precedent to the present paper's focus; however, it does not provide a framework for designing these documents. The present survey and taxonomy provide that missing design framework.\u003c/p\u003e \u003c/div\u003e"},{"header":"3 Research Questions","content":"\u003cp\u003eThis survey is organised around three explicit research questions, following the methodological convention for structured literature reviews (Kitchenham and Charters \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2007\u003c/span\u003e; Petersen et al. \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2008\u003c/span\u003e):\u003c/p\u003e \u003cp\u003e \u003cb\u003eRQ1.\u003c/b\u003e \u003cem\u003eWhat is the role of instruction documents in current agentic AutoML and autonomous research systems, and what design dimensions do they expose?\u003c/em\u003e\u003c/p\u003e \u003cp\u003e \u003cb\u003eRQ2.\u003c/b\u003e \u003cem\u003eWhat are the theoretically distinct families of instruction strategies, and what do they predict about agent search behaviour, exploration\u0026ndash;exploitation balance, and session-level outcomes?\u003c/em\u003e\u003c/p\u003e \u003cp\u003e \u003cb\u003eRQ3.\u003c/b\u003e \u003cem\u003eWhat empirical evidence supports the predicted effects of different instruction strategies, and how does this evidence validate the taxonomy across systems?\u003c/em\u003e\u003c/p\u003e \u003cp\u003eThese questions collectively address the absence of a principled framework for instruction document design in autonomous experimentation systems.\u003c/p\u003e"},{"header":"4 Survey of Autonomous Research and Agentic AutoML Systems","content":"\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Survey Methodology\u003c/h2\u003e \u003cp\u003eWe surveyed systems published or released between 2023 and March 2026 satisfying three criteria: (a) an LLM agent autonomously modifies code or experimental configurations; (b) modifications are evaluated against a scalar metric without human intervention; and (c) the agent iterates over multiple cycles within a session. Systems were identified through arXiv searches on 'autonomous ML research', 'agentic AutoML', 'LLM-driven code optimisation', and 'AI research agent', supplemented by forward citation tracking from AIDE (Jiang et al. \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) and backward citation tracking from AIRS-Bench (Lupidi et al. \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2026\u003c/span\u003e), and by reviewing related-work sections of AIRA (Toledo et al. \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2025\u003c/span\u003e), R\u0026amp;D-Agent (Yang et al. \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e2025\u003c/span\u003e), and MLAgentBench (Huang et al. \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eSixteen systems satisfied all criteria. Taxonomy development by a single author lacks inter-rater reliability; we address this by requiring each strategy family to be independently corroborated by at least two systems beyond the anchoring case (Sect. \u003cspan refid=\"Sec22\" class=\"InternalRef\"\u003e6.3\u003c/span\u003e). Saturation was assessed by checking whether the three systems added in the final pass (R\u0026amp;D-Agent, MLGym, AutoKaggle) introduced design choices not representable by the seven-dimension framework; they did not. Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e summarises results.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCross-system survey of instruction documents in agentic AutoML and autonomous research systems (16 systems, 2023\u0026ndash;March 2026)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSystem\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eInstruction Document\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eKey Design Dimensions\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSearch Mechanism\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eEvaluation Metric\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eautoresearch(Karpathy \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2026\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eprogram.md (Markdown)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAll 7 dimensions (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e); richest across systems\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eGreedy hill-climbing\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eval_bpb\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAIDE(Jiang et al. \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTask specification (text)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eScope, hypothesis hints, safety constraints\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTree search: Draft/Improve/Debug\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMLE-bench medal rate (16.9% o1-preview)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAIRA(Toledo et al. \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2025\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAIRA-dojo task spec (YAML+text)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eScope, operator set, search policy, exploration budget\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eConfigurable: Greedy/MCTS/Evolutionary\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMLE-bench medal rate\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eR\u0026amp;D-Agent(Yang et al. \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e2025\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eResearch task brief (text)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eScope, hypothesis seeding, exploration directive\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eDual-agent: Researcher\u0026thinsp;+\u0026thinsp;Developer, parallel traces\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMLE-bench medal rate (35.1% \u0026mdash; SOTA)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAgentHPO(Liu et al. \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2024a\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHPO task description\u0026thinsp;+\u0026thinsp;rationales\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHypothesis list, mechanistic rationales, prior experiment summaries\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSequential LLM Bayesian-style reasoning\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eTask accuracy/RMSE (12 benchmarks)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMLAgentBench(Huang et al. \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTask description (text\u0026thinsp;+\u0026thinsp;starter files)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTask objectives, allowed operations, evaluation protocol\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eReAct-style: read/write/execute code\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eTask success rate (37.5% Claude Opus \u0026mdash; SOTA at time)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAlphaEvolve(Novikov et al. \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2025\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eProblem specification (text)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eScope, correctness requirements, diversity hints\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eEvolutionary: Flash (volume)\u0026thinsp;+\u0026thinsp;Pro (depth)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCorrectness\u0026thinsp;+\u0026thinsp;efficiency\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAI-Researcher(Tang et al. \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e2025\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eResearch direction brief\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eDomain scope, resource filters, idea evaluation criteria\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eLiterature-driven idea generation pipeline\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eNovelty\u0026thinsp;+\u0026thinsp;feasibility scores\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAI Scientist(Lu et al. \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2024\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eResearch topic\u0026thinsp;+\u0026thinsp;templates\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTopic constraints, template, review criteria\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eFull pipeline: ideas\u0026rarr;experiments\u0026rarr;paper\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eReviewer score simulation\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAI Scientist-v2(Yamada et al. \u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e2025\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eWorkshop task description\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eResearch scope, milestone structure, collaboration roles\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAgentic tree search\u0026thinsp;+\u0026thinsp;sub-agents\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eWorkshop acceptance criteria\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAIRS-Bench(Lupidi et al. \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2026\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eproject_description.md (per task)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eScope, baseline info, allowed modifications, metric target\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAIRA-dojo or MLGym harness\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eNormalised score vs. SOTA\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAgent Lab(Schmidgall et al. \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2025\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eResearch task specification\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLiterature scope, experiment plan, sub-agent roles\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMulti-agent literature\u0026thinsp;+\u0026thinsp;experiment pipeline\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eBenchmark performance\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSELA(Chi et al. \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2024\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eML task description\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePipeline components, search depth\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMCTS over full AutoML pipeline\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eValidation accuracy\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFunSearch(Romera-Paredes et al. \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2023\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFunction spec (skeleton\u0026thinsp;+\u0026thinsp;docstring)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFunction interface, correctness, diversity targets\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eEvolutionary: LLM mutation\u0026thinsp;+\u0026thinsp;evaluator\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCombinatorial objective\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAutoKaggle(Li et al. \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2024b\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCompetition description (parsed)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFeature scope, model selection, validation strategy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMulti-agent data science pipeline\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eKaggle leaderboard score\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMLGym(Nathani et al. \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2025\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTask description file (text\u0026thinsp;+\u0026thinsp;code)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eResearch objectives, allowed methods, evaluation protocol, baseline reference\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eReAct-style LLM agent; RL-trainable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eTask-specific benchmark score\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Design Dimensions of Instruction Documents\u003c/h2\u003e \u003cp\u003eAcross all sixteen systems, the instruction document translates the practitioner's intent, knowledge, and constraints into agent-actionable form. Seven design dimensions appear consistently. Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e summarises these dimensions, their presence, and the range of choices observed.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eDesign dimensions of instruction documents across sixteen surveyed systems. Presence: H\u0026thinsp;=\u0026thinsp;high (explicitly supported), M\u0026thinsp;=\u0026thinsp;moderate (partially supported), L\u0026thinsp;=\u0026thinsp;low (implicit or absent)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDesign Dimension\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDescription and Range of Choices\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePresence (16 Systems)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModification scope\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eWhich components the agent may alter. Range: unconstrained to component-specific with explicit exclusion zones.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eH in all 16\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHypothesis seeding\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eWhether starting hypotheses are provided and with what supporting detail. Range: no seeding to ranked hypothesis trees with mechanistic rationales.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eH in 10; M in 6\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSimplicity criterion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eWhether complexity carries a penalty alongside metric performance. Range: pure metric maximisation to hard complexity veto above threshold.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eH in 5; M in 4; L in 7\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCross-session memory\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eWhether prior session findings are injected into the current instruction document. Range: none to full injection of prior discoveries as starting configurations.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eH in 3; M in 6; L in 7\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExploration directive\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eExplicit instruction on exploration\u0026ndash;exploitation balance. Range: implicit to hard category cap per N experiments with explicit exploration budget per phase.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eH in 5; M in 6; L in 5\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCurriculum staging\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eWhether the document specifies different strategies for different session phases. Range: uniform throughout to three-phase with convergence-triggered transitions.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eH in 3; M in 4; L in 9\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSafety constraints\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHard limits protecting session integrity. Range: soft resource limit only to hard parameter bounds with architectural invariants.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eH in 8; M in 5; L in 3\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e4.3 The autoresearch Framework as Anchoring Case\u003c/h2\u003e \u003cp\u003eAmong the sixteen surveyed systems, \u003cem\u003eautoresearch\u003c/em\u003e (Karpathy \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) exposes the richest set of design dimensions and its minimalist architecture makes program.md effects maximally visible. The system consists of a single approximately 630-line training script (train.py) modified freely by the agent; an immutable evaluation harness (prepare.py); and a human-authored Markdown instruction document (program.md). Every training run is fixed at exactly five minutes of wall time, making val_bpb\u0026mdash;validation bits-per-byte, a vocabulary-size-independent language modelling metric\u0026mdash;directly comparable across all experiments. On a single NVIDIA H100 GPU, approximately 12 experiments per hour are achievable, yielding approximately 100 experiments during a typical overnight session.\u003c/p\u003e \u003cp\u003eThe agent operates in a greedy hill-climbing loop: read program.md, propose a modification to train.py, execute the run, commit if val_bpb improved, revert if not. Because the agent has no other persistent state and every experimental outcome is numerically comparable, differences in session-level performance between program variants can be attributed to instruction design with unusually high confidence relative to more complex multi-component systems.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e4.4 Answer to RQ1\u003c/h2\u003e \u003cp\u003eAcross all sixteen surveyed systems, the instruction document is the practitioner's primary control mechanism. Seven design dimensions are common, though present to varying degrees. The design space is large, combinatorial, and currently navigated ad hoc. The taxonomy in Sect. \u003cspan refid=\"Sec12\" class=\"InternalRef\"\u003e5\u003c/span\u003e characterises the theoretically distinct strategy families that emerge from different configurations of these dimensions.\u003c/p\u003e \u003c/div\u003e"},{"header":"5 Taxonomy of Instruction Strategy Families","content":"\u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e5.1 Taxonomy Development Method\u003c/h2\u003e \u003cp\u003eThe taxonomy was developed through structured content analysis of five inputs: (1) the reference \u003cem\u003eautoresearch\u003c/em\u003e program.md and all publicly documented session variants (Karpathy \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2026\u003c/span\u003e); (2) the cross-system survey in Sect. \u003cspan refid=\"Sec7\" class=\"InternalRef\"\u003e4\u003c/span\u003e; (3) community fork activity and program variants in the \u003cem\u003eautoresearch\u003c/em\u003e GitHub Discussions; (4) design choices documented in AIDE, AIRA, R\u0026amp;D-Agent, AgentHPO, AlphaEvolve, MLAgentBench, AI-Researcher, and AIRS-Bench; and (5) theoretical frameworks from the prompt engineering (Sclar et al. \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Renduchintala et al. \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Zhuo et al. \u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), AutoML (Snoek et al. \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2012\u003c/span\u003e; Wistuba et al. \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e2015\u003c/span\u003e), evolutionary computation (B\u0026auml;ck et al. \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; van Stein and B\u0026auml;ck \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), curriculum learning (Bengio et al. \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2009\u003c/span\u003e), and population-based training (Jaderberg et al. \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2017\u003c/span\u003e) literatures.\u003c/p\u003e \u003cp\u003eA strategy family was defined as a coherent configuration of dimension choices that: (a) appears repeatedly across sources; (b) is theoretically distinguishable from other families along at least two analytical dimensions; and (c) has direct empirical support or a clear theoretical prediction from the related literature. Five families resulted. They are not mutually exclusive\u0026mdash;effective programs typically combine elements of two or three families. Each family is characterised along four dimensions: (i) exploration\u0026ndash;exploitation balance; (ii) modification emphasis; (iii) primary failure mode; and (iv) conditions of applicability. Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e summarises all five families.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003e5.2 Scope-Constrained Strategy\u003c/h2\u003e \u003cp\u003eA Scope-Constrained program explicitly restricts permissible modifications to a specified system component\u0026mdash;the attention mechanism, the optimiser, positional encoding, or any other named subsystem. The rationale is search efficiency through dimensionality reduction: constraining scope increases the density of productive experiments per unit of compute, directly analogous to search space pruning in classical AutoML (Wistuba et al. \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e2015\u003c/span\u003e). Where classical pruning is algorithmic, scope-constraining is linguistic.\u003c/p\u003e \u003cp\u003eIn AIRA terminology (Toledo et al. \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2025\u003c/span\u003e), this corresponds to restricting the operator set to a component-specific subset. In AIRS-Bench (Lupidi et al. \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2026\u003c/span\u003e), the project_description.md files instantiate this pattern. In R\u0026amp;D-Agent (Yang et al. \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e2025\u003c/span\u003e), the research task brief scopes the Researcher agent's hypothesis space. In MLAgentBench (Huang et al. \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), task descriptions constrain the allowable code operations.\u003c/p\u003e \u003cp\u003e \u003cb\u003eExploration\u0026ndash;exploitation balance.\u003c/b\u003e Strongly exploitative; all session compute is directed toward depth within the constrained component.\u003c/p\u003e \u003cp\u003e \u003cb\u003eModification emphasis.\u003c/b\u003e Component-specific architectural adjustments or hyperparameter tuning within the named subsystem.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePrimary failure mode.\u003c/b\u003e Premature convergence if the constrained component lacks optimisation headroom. Ill-suited to early exploratory sessions where the productive region is unknown.\u003c/p\u003e \u003cp\u003e \u003cb\u003eConditions of applicability.\u003c/b\u003e Most effective when prior evidence establishes that a specific component is under-optimised, or as the second phase of a curriculum-staged session.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e5.3 Hypothesis-Directed Strategy\u003c/h2\u003e \u003cp\u003eA Hypothesis-Directed program provides structured testable hypotheses, each paired with a mechanistic rationale explaining the expected effect under the session's specific constraints. For example: \"Halving total batch size will increase gradient step count within the 5-minute budget by approximately 40%; the training signal remains stable above approximately 32K tokens per step at this model scale, so the additional gradient steps should translate to improved val_bpb.\" Pairing rationales with hypotheses engages the agent's chain-of-thought reasoning capabilities (Wei et al. \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e2022\u003c/span\u003e), improving the quality of code modifications.\u003c/p\u003e \u003cp\u003eAgentHPO (Liu et al. \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2024a\u003c/span\u003e) validates this strategy under controlled conditions. Across twelve benchmark HPO tasks, agents provided with mechanistic rationales reached competitive performance with fewer function evaluations than baseline agents operating without structured scaffolding\u0026mdash;the paper reports improvements across all twelve tasks, with the hypothesis-directed agent requiring roughly half the number of trials to match the best configurations found by the baseline in the median case. R\u0026amp;D-Agent (Yang et al. \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) independently validates this at system level: its explicit researcher\u0026ndash;developer separation, where the Researcher agent constructs mechanistic hypotheses before implementation, is credited as the key driver of its state-of-the-art 35.1% MLE-bench medal rate, compared to 16.9% for AIDE without this mechanism. DS-Agent (Guo et al. \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) provides further corroboration via case-based reasoning provision.\u003c/p\u003e \u003cp\u003e \u003cb\u003eExploration\u0026ndash;exploitation balance.\u003c/b\u003e Focused exploitation of practitioner domain knowledge. The instruction should authorise the agent to generate its own hypotheses as the list is exhausted, preventing anchoring bias.\u003c/p\u003e \u003cp\u003e \u003cb\u003eModification emphasis.\u003c/b\u003e Hypothesis-driven modifications within the components addressed by the provided rationales.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePrimary failure mode.\u003c/b\u003e Anchoring bias\u0026mdash;excessive concentration on the provided list even when early experimental evidence indicates those hypotheses are unproductive.\u003c/p\u003e \u003cp\u003e \u003cb\u003eConditions of applicability.\u003c/b\u003e Most effective for refinement sessions where the practitioner has strong domain theory about which modifications are likely to improve the metric and why.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003e5.4 Diversity-Preserving Strategy\u003c/h2\u003e \u003cp\u003eA Diversity-Preserving program explicitly instructs the agent to maintain variety across modification categories\u0026mdash;capping consecutive experiments within any single category and mandating rotation across predefined categories. This counteracts the hyperparameter-fixation failure mode of greedy hill-climbing, which is structurally analogous to exploration collapse in RL-based agent self-evolution (Wang et al. \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). The population-based training literature (Jaderberg et al. \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2017\u003c/span\u003e) provides the theoretical foundation. This principle is directly connected to diversity maintenance in evolutionary algorithms (B\u0026auml;ck et al. \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2023\u003c/span\u003e)\u0026mdash;population diversity in evolutionary computation corresponds to temporal modification diversity in a single-agent session.\u003c/p\u003e \u003cp\u003eAlphaEvolve (Novikov et al. \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) instantiates this at system level through its dual-model architecture (Gemini Flash for high-volume exploration, Gemini Pro for selective refinement); FunSearch (Romera-Paredes et al. \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) maintains island-based populations with explicit diversity preservation. MLAgentBench (Huang et al. \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) found that agents frequently fail by fixating on initial strategies without systematic exploration\u0026mdash;precisely the failure mode this strategy addresses. Community-reported observations from the Hyperspace distributed run (McKinney \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) suggest hardware-constrained diversity across 35 agents produced non-redundant discoveries; this is cited as qualitative corroboration only.\u003c/p\u003e \u003cp\u003e \u003cb\u003eExploration\u0026ndash;exploitation balance.\u003c/b\u003e Explicitly exploratory; the instruction mandates coverage breadth.\u003c/p\u003e \u003cp\u003e \u003cb\u003eModification emphasis.\u003c/b\u003e Balanced coverage across all modification categories, with equal attention to architectural, optimiser, training dynamics, and initialisation changes.\u003c/p\u003e \u003cp\u003e \u003cb\u003eModification emphasis.\u003c/b\u003e Balanced coverage across all modification categories, with equal attention to architectural, optimiser, training dynamics, and initialisation changes.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePrimary failure mode.\u003c/b\u003e Time fragmentation if the per-category cap is set too tightly for the session length.\u003c/p\u003e \u003cp\u003e \u003cb\u003eConditions of applicability.\u003c/b\u003e Most appropriate for early-stage exploratory sessions, or when prior sessions have converged to a suspected local optimum.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003e5.5 Simplicity-Biased Strategy\u003c/h2\u003e \u003cp\u003eThe reference \u003cem\u003eautoresearch\u003c/em\u003e program.md (Karpathy \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) includes an explicit simplicity criterion: modifications producing marginal metric improvements accompanied by significant code complexity are not retained; simplifications yielding equal or better performance are actively rewarded. The practical justification is transferability. GitHub Discussions (Karpathy \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2026\u003c/span\u003e, Discussion #43) document that approximately 20 \u003cem\u003eautoresearch\u003c/em\u003e-discovered improvements transferred from depth-12 to depth-24 models; secondary reporting (Schmid \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) indicates an 11% speedup at depth-24. These transfer claims require independent replication but are directionally consistent with the theoretical prediction that simpler modifications carry fewer context-specific assumptions.\u003c/p\u003e \u003cp\u003eAlphaEvolve's discovery of the 48-scalar-multiplication matrix multiplication algorithm (Novikov et al. \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) and FunSearch's cap-set discovery (Romera-Paredes et al. \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) demonstrate that simplicity-biased evolutionary search can yield results of genuine scientific significance\u0026mdash;both algorithms are simultaneously the highest-performing known and structurally the most parsimonious.\u003c/p\u003e \u003cp\u003e \u003cb\u003eExploration\u0026ndash;exploitation balance.\u003c/b\u003e Neutral; biases exploitation within the discovered region toward simpler solutions.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePrimary failure mode.\u003c/b\u003e Suppression of high-gain complex modifications. Mitigation: specify simplicity acts as a tiebreaker, not a veto, above an explicit threshold.\u003c/p\u003e \u003cp\u003e \u003cb\u003eConditions of applicability.\u003c/b\u003e Whenever discovered optimisations are expected to transfer across model scales, architectures, or tasks.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003e5.6 Curriculum-Staged Strategy\u003c/h2\u003e \u003cp\u003eA Curriculum-Staged program divides the session into phases with distinct strategic objectives, using convergence-based conditions to trigger transitions. The theoretical motivation draws on curriculum learning (Bengio et al. \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2009\u003c/span\u003e). AIRA (Toledo et al. \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) provides direct quantitative support: pure exploitation is dominated by more exploratory policies in sessions exceeding approximately 40 experiments, and the performance gap widens with session length. This is directly analogous to the finding in MLAgentBench (Huang et al. \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) that agents struggle with long-term planning\u0026mdash;curriculum staging explicitly addresses this by structuring the long-horizon plan into phases with clear objectives and transition conditions.\u003c/p\u003e \u003cp\u003eA three-phase structure is proposed as a calibration starting point: Phase 1 (coarse exploration, \u0026asymp;\u0026thinsp;30% of experiments)\u0026mdash;wide category coverage, accepting improvements\u0026thinsp;\u0026gt;\u0026thinsp;0.0005 val_bpb, transitioning when fewer than 20% of the last ten experiments improve; Phase 2 (targeted refinement, \u0026asymp;\u0026thinsp;50%)\u0026mdash;focus on two to three most productive categories, threshold 0.0002; Phase 3 (robustness verification, \u0026asymp;\u0026thinsp;20%)\u0026mdash;conservative threshold 0.0001. These thresholds are calibration starting points requiring empirical validation. AI Scientist-v2 (Yamada et al. \u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) independently instantiates this principle with progress-triggered phase transitions.\u003c/p\u003e \u003cp\u003e \u003cb\u003eExploration\u0026ndash;exploitation balance.\u003c/b\u003e Dynamically adaptive: E\u0026rarr;X\u0026rarr;conservative.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePrimary failure mode.\u003c/b\u003e Mis-timed phase transitions.\u003c/p\u003e \u003cp\u003e \u003cb\u003eConditions of applicability.\u003c/b\u003e Sessions of 60 or more experiments.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSummary of the five instruction strategy families. E\u0026ndash;E\u0026thinsp;=\u0026thinsp;exploration\u0026ndash;exploitation balance; E\u0026thinsp;=\u0026thinsp;exploratory, X\u0026thinsp;=\u0026thinsp;exploitative. Families are not mutually exclusive\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFamily\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eE\u0026ndash;E Balance\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCore Mechanism\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePrimary Failure Mode\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eBest Applied When\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eScope-Constrained\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStrongly X\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLinguistic scope restriction reduces search dimensionality\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePremature convergence without headroom\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eComponent known under-optimised\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHypothesis-Directed\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFocused X\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMechanistic rationales trigger chain-of-thought\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAnchoring bias\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eStrong practitioner domain theory\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDiversity-Preserving\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eE (mandated)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCategory-rotation prevents hyperparameter fixation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTime fragmentation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eEarly-stage; suspected local optimum\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSimplicity-Biased\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNeutral; biases within region\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eComplexity penalty\u0026thinsp;+\u0026thinsp;metric tiebreaker; deletions valued\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSuppression of high-gain complex changes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eTransferability or interpretability required\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCurriculum-Staged\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAdaptive E\u0026rarr;X\u0026rarr;conservative\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eConvergence-triggered phase transitions\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMis-timed transitions\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSessions\u0026thinsp;\u0026ge;\u0026thinsp;60 experiments\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"6 Empirical Grounding and Cross-System Validation","content":"\u003cdiv id=\"Sec20\" class=\"Section2\"\u003e \u003ch2\u003e6.1 Autoresearch Session Analysis\u003c/h2\u003e \u003cp\u003eThe two publicly documented \u003cem\u003eautoresearch\u003c/em\u003e sessions (Karpathy \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) provide directly interpretable evidence. The architecture makes program.md effects analytically visible: hardware, evaluation metric, and agent LLM are held constant within each session, and every outcome is recorded with a numerical val_bpb score.\u003c/p\u003e \u003cp\u003eBefore presenting the session comparison, the following methodological caveat is essential: Sessions A and B differ on two variables simultaneously. First, the instruction document content\u0026mdash;Session B injected Session A's top discoveries as starting priors. Second, session length\u0026mdash;Session A ran 89 experiments, Session B ran 126, a 42% difference. On a holistic per-experiment basis, Session B was marginally \u003cem\u003eless\u003c/em\u003e efficient than Session A (0.0282/126\u0026thinsp;=\u0026thinsp;0.000224 per experiment versus 0.0206/89\u0026thinsp;=\u0026thinsp;0.000231 per experiment). The 37% total gain advantage of Session B is thus consistent with being explained by the additional 42% of experiments alone, if per-experiment efficiency were constant. This session-length confound prevents clean causal attribution of total gain differences to the instruction intervention.\u003c/p\u003e \u003cp\u003eWith this caveat stated, the session data contains a pattern that session length alone cannot account for: the qualitative composition of what was discovered. Session A (Discussion #32, 89 experiments) improved val_bpb from 0.9979 to 0.9773 without cross-session priors; its primary discoveries were batch halving, depth 9, SSSSL window pattern, and RoPE 200K. Session B (Discussion #43, 126 experiments) achieved total gain 0.0282; its primary discoveries were weight decay on embedding and value projection parameters, and transformer initialisation scale. Session A had 89 experiments to discover weight decay strategies and did not; Session B found them within the first 30 experiments after prior injection. This qualitative compositional difference is not explained by session length and is consistent with the Curriculum-Staged family's cross-session memory mechanism. Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e summarises metrics.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparative metrics for Sessions A and B (Karpathy \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2026\u003c/span\u003e). The 37% total gain difference is length-confounded and must not be interpreted as a causal estimate of the instruction intervention effect\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMetric\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSession A\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSession B\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eInterpretation\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTotal experiments\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e89\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e126\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e+\u0026thinsp;42% more in B \u0026mdash; confound\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eStarting val_bpb\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.9979\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.9979\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eIdentical baseline\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTotal val_bpb gain\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.0206\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.0282\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e+\u0026thinsp;37% in B; length-confounded (see text)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHolistic per-experiment efficiency\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.000231\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.000224\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eB marginally less \u0026mdash; length alone could explain total gap\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperiments to first 0.010 gain\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e~\u0026thinsp;15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e~\u0026thinsp;5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e67% reduction \u0026mdash; exploratory phase compressed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLate-phase efficiency (exp 16+; post-hoc window)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e~\u0026thinsp;0.000089\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e~\u0026thinsp;0.00021\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2.36\u0026times; (post-hoc segmentation; no uncertainty estimate)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePrimary discoveries\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBatch halving; depth 9; SSSSL; RoPE 200K\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eWD on embeddings; init scale 0.68\u0026times;\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eQualitatively different \u0026mdash; not explained by length alone\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec21\" class=\"Section2\"\u003e \u003ch2\u003e6.2 Evidence from the Agentic AutoML Literature\u003c/h2\u003e \u003cp\u003eThe peer-reviewed literature provides controlled experimental support for the taxonomy's predictions. AgentHPO (Liu et al. \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2024a\u003c/span\u003e) directly validates the Hypothesis-Directed family under controlled conditions: across twelve benchmark HPO tasks, agents with mechanistic rationales required significantly fewer trials than unguided baseline agents. Specifically, the paper reports that hypothesis-directed agents reached competitive performance in roughly half the number of evaluations compared to the baseline in the median case across the twelve tasks, with improvements observed on all tasks. The mechanism\u0026mdash;chain-of-thought priming (Wei et al. \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e2022\u003c/span\u003e)\u0026mdash;operates at every agent call, priming the model to reason about expected experimental outcomes before executing modifications. R\u0026amp;D-Agent (Yang et al. \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) independently validates this at system level: its 35.1% MLE-bench medal rate compared to AIDE's 16.9% (Chan et al. \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) provides controlled evidence that structured hypothesis separation (Researcher vs. Developer role) roughly doubles system performance.\u003c/p\u003e \u003cp\u003eAIRA (Toledo et al. \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) directly validates the Curriculum-Staged family. Their systematic comparison of Greedy, MCTS, and Evolutionary policies across MLE-bench tasks demonstrates that pure exploitation is dominated by more exploratory policies in sessions exceeding approximately 40 experiments, and the performance gap widens with session length. AIRA also documents 9\u0026ndash;13 percentage-point performance gaps from validation-set overfitting in long sessions. MLAgentBench (Huang et al. \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) provides independent corroboration of the long-term planning challenge: success rate drops from 100% on short, well-defined tasks to 0\u0026ndash;25% on open-ended Kaggle challenges, consistent with the prediction that undifferentiated uniform strategies fail in extended sessions.\u003c/p\u003e \u003cp\u003eAlphaEvolve (Novikov et al. \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) and FunSearch (Romera-Paredes et al. \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) jointly validate the Simplicity-Biased family. The Diversity-Preserving family is supported by AIRA's finding that evolutionary policies outperform greedy in extended sessions and by population-based training theory (Jaderberg et al. \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2017\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec22\" class=\"Section2\"\u003e \u003ch2\u003e6.3 Cross-System Taxonomy Validation\u003c/h2\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e maps each family to supporting systems, distinguishing explicit instantiation from implicit corroboration.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCross-system validation of the five taxonomy families. Explicit\u0026thinsp;=\u0026thinsp;family explicitly instantiated in system's design; Implicit\u0026thinsp;=\u0026thinsp;effects observable but not named as such\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFamily\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSupporting Systems\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNature of Evidence\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eScope-Constrained\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAIRS-Bench; AIDE; R\u0026amp;D-Agent; MLAgentBench\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eExplicit in AIRS-Bench task definitions and R\u0026amp;D-Agent task briefs. Implicit in AIDE and MLAgentBench task specifications. autoresearch data: tenfold efficiency decline consistent with absence of scope constraint.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHypothesis-Directed\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAgentHPO; R\u0026amp;D-Agent; AI-Researcher; DS-Agent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eQuantified and controlled in AgentHPO: structured hypothesis provision reduces trials across 12 HPO tasks (Explicit). R\u0026amp;D-Agent: researcher\u0026ndash;developer separation credited as key driver (35.1% vs. 16.9% medal rate). AI-Researcher Idea Generator module (Explicit). DS-Agent: case-based reasoning (equivalent mechanism).\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDiversity-Preserving\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAlphaEvolve; FunSearch; AIRA (evolutionary); MLAgentBench\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eExplicit diversity mechanisms in AlphaEvolve (dual-model) and FunSearch (island populations). AIRA: evolutionary policy outperforms greedy in extended sessions (controlled). MLAgentBench: agent strategy fixation identified as primary failure mode. Hyperspace observations: qualitative corroboration only (McKinney \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2026\u003c/span\u003e).\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSimplicity-Biased\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eautoresearch; AlphaEvolve; FunSearch; AI Scientist\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eExplicit in reference program.md simplicity criterion. Transfer evidence from GitHub Discussions (Karpathy \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) and Schmid (\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) \u0026mdash; requires independent replication. AlphaEvolve: 48-scalar matrix multiplication (highest-performing\u0026thinsp;+\u0026thinsp;structurally simplest). FunSearch: cap-set discovery.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCurriculum-Staged\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAIRA; AI Scientist-v2; autoresearch; Agent Lab; MLAgentBench\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eQuantified and controlled in AIRA: greedy dominated by exploratory policies\u0026thinsp;\u0026gt;\u0026thinsp;40 experiments; gap widens with length. MLAgentBench: agent success drops on long-horizon tasks (0\u0026ndash;25%), validating long-session planning challenge. AI Scientist-v2: phase-structured pipeline. autoresearch: qualitative territory change (length-confounded for total gain).\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec23\" class=\"Section2\"\u003e \u003ch2\u003e6.4 Quantitative Performance Landscape\u003c/h2\u003e \u003cp\u003eTo contextualise the practical significance of instruction design, Table\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e6\u003c/span\u003e summarises quantitative performance benchmarks across systems for which standardised results are available. This table illustrates the performance gap across system designs\u0026mdash;a gap that the present paper argues is substantially determined by the quality of instruction documents, particularly the Hypothesis-Directed and Scope-Constrained design choices that distinguish high-performing systems.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab6\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 6\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eQuantitative performance overview of systems with available MLE-bench or standardised benchmark results. All MLE-bench figures are 'any medal rate' (%). MLAgentBench figures are task success rate (%). Results reflect best published configuration at time of survey (March 2026).\u003c/p\u003e \u003cdiv class=\"Credit\"\u003e\u003cp\u003eSources: Chan et al. (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), Toledo et al. (\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2025\u003c/span\u003e), Yang et al. (\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e2025\u003c/span\u003e), Huang et al. (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e)\u003c/p\u003e\u003c/div\u003e\u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSystem\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBenchmark\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eBest Published Result\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eKey Instruction Feature\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eNotes\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eR\u0026amp;D-Agent (Yang et al. \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e2025\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMLE-bench\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e35.1% any medal\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eExplicit hypothesis\u0026ndash;implementation separation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSOTA open-source at survey cutoff; dual-agent design\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAIRA \u0026mdash; Evolutionary (Toledo et al. \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2025\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMLE-bench\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u0026gt;AIDE greedy in sessions\u0026thinsp;\u0026gt;\u0026thinsp;40 exp\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eDiversity-preserving evolutionary search policy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eControlled policy comparison; gap widens with length\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAIDE\u0026thinsp;+\u0026thinsp;o1-preview (Jiang et al. \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMLE-bench\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e16.9% any medal\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eGreedy tree search; no explicit hypothesis seeding\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eBaseline for comparison; doubles to 34.1% at pass@8\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAgentHPO (Liu et al. \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2024a\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHPO (12 tasks)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e~\u0026thinsp;50% fewer trials vs. baseline\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHypothesis-directed with mechanistic rationales\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eControlled study; improvements across all 12 tasks\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMLAgentBench\u0026mdash;Claude Opus (Huang et al. \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMLAgentBench\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e37.5% task success (best agent)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eReAct-style; no structured curriculum\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0% on recent Kaggle tasks; 100% on simple tasks\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAlphaEvolve (Novikov et al. \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2025\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAlgorithm discovery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e48-scalar matmul; 0.7% DC efficiency\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eDiversity-preserving dual-model; simplicity bias\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eScientific significance: new mathematical result\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe quantitative landscape in Table\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e6\u003c/span\u003e reveals a clear pattern: systems with explicit hypothesis-directed and diversity-preserving instruction mechanisms outperform those relying on undifferentiated greedy search. The 35.1% vs. 16.9% medal rate gap between R\u0026amp;D-Agent and AIDE on identical benchmarks\u0026mdash;with the key architectural difference being the Hypothesis-Directed design of R\u0026amp;D-Agent's Researcher component\u0026mdash;provides the clearest available controlled evidence that instruction strategy design has measurable, substantial impact on system performance.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec24\" class=\"Section2\"\u003e \u003ch2\u003e6.5 Validation-Set Overfit Risk\u003c/h2\u003e \u003cp\u003eAny iterative optimisation system re-querying a fixed validation set risks progressive overfit. AIRA (Toledo et al. \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) quantifies this: selecting the final configuration by held-out test-set score rather than validation score improved performance by 9\u0026ndash;13 percentage points across MLE-bench tasks. Instruction design can mitigate this risk: the Curriculum-Staged Phase 3 conservative acceptance threshold limits configurations accumulating marginal overfit gains; the Simplicity-Biased strategy's preference for architectural modifications reduces the surface area available for validation-set exploitation.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec25\" class=\"Section2\"\u003e \u003ch2\u003e6.6 Answer to RQ3\u003c/h2\u003e \u003cp\u003eMulti-source evidence corroborates all five taxonomy families. The strongest controlled evidence comes from AgentHPO for the Hypothesis-Directed family and AIRA for the Curriculum-Staged family. The quantitative performance landscape in Table\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e6\u003c/span\u003e demonstrates that systems implementing Hypothesis-Directed and Diversity-Preserving instruction mechanisms achieve substantially higher benchmark performance than systems without these features. The \u003cem\u003eautoresearch\u003c/em\u003e session comparison provides suggestive but non-causal evidence; the session-length confound prevents attribution of the 37% total gain difference to instruction design alone.\u003c/p\u003e \u003c/div\u003e"},{"header":"7 Practitioner Guidelines","content":"\u003cp\u003eFive guidelines are synthesised for practitioners authoring instruction documents. All numerical thresholds are calibration starting points requiring empirical validation in specific deployment contexts.\u003c/p\u003e \u003cp\u003e \u003cb\u003eGuideline 1 \u0026mdash; Inject cross-session priors before any follow-on run.\u003c/b\u003e Update program.md to include all discoveries from prior sessions as established starting configurations before beginning a second or subsequent session. This is the highest-leverage intervention suggested by the available evidence: it compresses the exploratory phase (approximately 67% reduction in experiments-to-first-categorical-gain in the \u003cem\u003eautoresearch\u003c/em\u003e sessions), redirects session attention toward qualitatively new territory, and is consistent with the Curriculum-Staged strategy's cross-session memory mechanism. The 37% total gain advantage attributed to this intervention is length-confounded (Sect. \u003cspan refid=\"Sec20\" class=\"InternalRef\"\u003e6.1\u003c/span\u003e) and should not be interpreted as a controlled causal estimate.\u003c/p\u003e \u003cp\u003e \u003cb\u003eGuideline 2 \u0026mdash; Stage the instruction for sessions of 60 or more experiments.\u003c/b\u003e Implement a two- or three-phase curriculum with convergence-rate transitions. Suggested calibration starting points: fewer than 20% improvement rate in the last ten experiments as the Phase 1-to-Phase 2 transition condition; rising acceptance thresholds in Phase 3. AIRA's controlled comparison (Toledo et al. \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) provides the motivation: greedy is dominated by exploratory policies in extended sessions. For sessions of 40 experiments or fewer, curriculum management overhead is unlikely to be worthwhile.\u003c/p\u003e \u003cp\u003e \u003cb\u003eGuideline 3 \u0026mdash; Pair every hypothesis with a mechanistic rationale.\u003c/b\u003e Rather than listing modifications to attempt, explain why each is expected to improve performance given the session's specific compute constraints. AgentHPO (Liu et al. \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2024a\u003c/span\u003e) demonstrates roughly 50% reduction in trials across 12 tasks; R\u0026amp;D-Agent (Yang et al. \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) demonstrates a more than doubling of MLE-bench medal rate compared to unstructured approaches (35.1% vs. 16.9%). As the session progresses, authorise the agent to generate its own rationale-equipped hypotheses, preventing anchoring bias.\u003c/p\u003e \u003cp\u003e \u003cb\u003eGuideline 4 \u0026mdash; Encode a category-rotation directive for exploratory sessions.\u003c/b\u003e Specify a cap of 8\u0026ndash;10 consecutive experiments within any single modification category for a 100-experiment session, with mandatory rotation across at least five distinct categories. This starting point (cap\u0026thinsp;\u0026asymp;\u0026thinsp;0.08\u0026ndash;0.10 \u0026times; total planned experiments) is motivated by the \u003cem\u003eautoresearch\u003c/em\u003e session data showing tenfold efficiency decline without a diversity directive and by MLAgentBench evidence (Huang et al. \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) of strategy fixation as a primary failure mode.\u003c/p\u003e \u003cp\u003e \u003cb\u003eGuideline 5 \u0026mdash; Calibrate the simplicity criterion explicitly.\u003c/b\u003e Specify that simplicity acts as a tiebreaker\u0026mdash;not a veto\u0026mdash;for metric-equivalent proposals. Suggested starting threshold: improvements exceeding 0.002 val_bpb are accepted regardless of added complexity; improvements below this threshold favour the simpler option. This threshold is illustrative and should be adjusted based on typical early-session gain magnitudes.\u003c/p\u003e"},{"header":"8 Limitations and Future Research Directions","content":"\u003cdiv id=\"Sec28\" class=\"Section2\"\u003e \u003ch2\u003e8.1 Limitations\u003c/h2\u003e \u003cp\u003eFive categories of limitation apply. First, the primary observational evidence involves only two \u003cem\u003eautoresearch\u003c/em\u003e sessions with no controlled random seed matching. The session-length confound (89 vs. 126 experiments) prevents clean causal attribution. All session comparison figures should be treated as descriptive statistics from non-randomised observations.\u003c/p\u003e \u003cp\u003eSecond, the taxonomy was developed inductively by a single author from a specific framework (LLM pretraining, 5-minute runs, single GPU). Single-author taxonomy development lacks the inter-rater reliability of multi-coder protocols. The saturation criterion and cross-system validation in Sect. \u003cspan refid=\"Sec22\" class=\"InternalRef\"\u003e6.3\u003c/span\u003e partially address this.\u003c/p\u003e \u003cp\u003eThird, agent behaviour depends substantially on the LLM backbone. The optimal instruction strategy for a reasoning-capable model (e.g., capable of strong chain-of-thought) likely differs from what is optimal for weaker models. The performance sensitivity identified by MLAgentBench (Huang et al. \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2024\u003c/span\u003e)\u0026mdash;ranging from 100% success on simple tasks to 0% on open-ended challenges\u0026mdash;illustrates that task characteristics also moderate instruction strategy effectiveness in ways our taxonomy does not fully capture.\u003c/p\u003e \u003cp\u003eFourth, some empirical claims rest on grey literature: the Simplicity-Biased transferability claim relies on GitHub Discussions and a secondary blog post (Schmid \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2026\u003c/span\u003e); the Diversity-Preserving Hyperspace evidence relies on media coverage (McKinney \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2026\u003c/span\u003e). These should be treated as directionally indicative until independently replicated.\u003c/p\u003e \u003cp\u003eFifth, the landscape is rapidly evolving. New systems have emerged since the survey cutoff, and the \u003cem\u003eautoresearch\u003c/em\u003e repository continues receiving community contributions. The taxonomy is a snapshot of a field in motion.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec29\" class=\"Section2\"\u003e \u003ch2\u003e8.2 Future Research Directions\u003c/h2\u003e \u003cp\u003e \u003cb\u003eControlled experimental comparison of program variants.\u003c/b\u003e The highest-priority next step is a controlled comparison of instruction strategies with matched random seeds, identical hardware, and identical session lengths. Even a 2\u0026times;2 factorial design (Diversity-Preserving vs. Scope-Constrained \u0026times; with vs. without cross-session priors) across ten matched session pairs would provide the first controlled causal evidence for the predictions this taxonomy makes.\u003c/p\u003e \u003cp\u003e \u003cb\u003eAutomated program optimisation.\u003c/b\u003e If instruction documents are the primary practitioner-facing control, optimising them is itself an optimisation problem. The prompt engineering literature provides gradient-free methods for automatic single-turn prompt optimisation (Li et al. \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2025\u003c/span\u003e); LLaMEA-style LLM evolutionary algorithms (van Stein and B\u0026auml;ck \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) and in-the-loop HPO for LLM-based heuristic design (van Stein et al. \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e2026\u003c/span\u003e) provide methodological foundations that could be extended to research program optimisation.\u003c/p\u003e \u003cp\u003e \u003cb\u003eStandardised instruction quality evaluation protocols.\u003c/b\u003e The field lacks a benchmark for instruction document quality comparable to MLE-bench (Chan et al. \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) for agent architecture quality. Developing a standardised evaluation protocol\u0026mdash;specifying matched session conditions, control programs, and quality metrics\u0026mdash;would enable systematic comparison of instruction strategies across the research community.\u003c/p\u003e \u003cp\u003e \u003cb\u003eMulti-agent coordination through complementary program assignments.\u003c/b\u003e Assigning complementary instruction strategies to coordinated agents would extend the Diversity-Preserving strategy from single-agent instruction design to multi-agent program design, building on insights from the Hyperspace distributed run (McKinney \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2026\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cb\u003eLLM-specific program design optimisation.\u003c/b\u003e The optimal program design likely varies with backbone LLM characteristics. A systematic study of strategy\u0026ndash;LLM interaction effects would allow the guidelines in Sect. \u003cspan refid=\"Sec26\" class=\"InternalRef\"\u003e7\u003c/span\u003e to be refined with model-specific calibrations.\u003c/p\u003e \u003c/div\u003e"},{"header":"9 Conclusions","content":"\u003cp\u003eThis paper has addressed a gap at the intersection of autonomous machine learning experimentation and LLM instruction engineering. In autonomous experimentation systems\u0026mdash;from the 16.9% medal-rate AIDE to the 35.1% medal-rate R\u0026amp;D-Agent to the 37.5% task-success MLAgentBench baseline\u0026mdash;the practitioner's primary lever is the instruction document that directs the agent across an extended session of dozens or hundreds of consecutive decisions. Yet this document has received almost no analytical attention as an engineering artefact.\u003c/p\u003e \u003cp\u003eFour contributions have been made. First, a structured survey of sixteen agentic AutoML and autonomous research systems\u0026mdash;positioned within a new related work section that situates the paper against existing surveys\u0026mdash;establishing that the instruction document is a universal practitioner-facing control mechanism, exposing seven design dimensions. Second, a five-family taxonomy grounded in theory from the AutoML, evolutionary computation (B\u0026auml;ck et al. \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; van Stein and B\u0026auml;ck \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), prompt engineering, and curriculum learning literatures, with each family characterised along four analytical dimensions and cross-validated against at least two independent systems. Third, multi-source empirical grounding that links the taxonomy's predictions to controlled experimental evidence: the 35.1% vs. 16.9% medal rate gap between R\u0026amp;D-Agent and AIDE\u0026mdash;attributable to hypothesis-directed instruction design\u0026mdash;provides the clearest available controlled evidence that instruction strategy has measurable, substantial impact on system performance. Fourth, five practitioner guidelines with explicitly labelled calibration thresholds, validated against all sixteen systems.\u003c/p\u003e \u003cp\u003eThe key finding is not merely that instruction document quality matters\u0026mdash;it is that principled, theoretically-grounded instruction strategy design explains a substantial fraction of the observed performance variation across current state-of-the-art systems. As the research community moves from individual system development toward benchmarked comparison on standardised tasks, understanding and engineering the instruction layer will become as important as engineering the search algorithm or the agent architecture. The taxonomy, performance landscape, and research agenda presented here provide the foundation for that work.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eFunding\u003c/p\u003e\n\u003cp\u003eThe author did not receive support from any organisation for the submitted work. No funds, grants, or other support was received.\u003c/p\u003e\n\u003cp\u003eCompeting Interests\u003c/p\u003e\n\u003cp\u003eThe author has no relevant financial or non-financial interests to disclose.\u003c/p\u003e\n\u003cp\u003eAuthor Contributions\u003c/p\u003e\n\u003cp\u003ePraneeth Kodumagulla: Conceptualization, Formal analysis, Investigation, Methodology, Writing\u0026mdash;original draft, Writing\u0026mdash;review and editing. The author has read and agreed to the submitted version of the manuscript.\u003c/p\u003e\n\u003cp\u003eData Availability\u003c/p\u003e\n\u003cp\u003eNo new primary datasets were created in this study. The \u003cem\u003eautoresearch\u003c/em\u003e session logs analysed in Sect. 6.1 are publicly available as GitHub Discussions #32 and #43 at https://github.com/karpathy/autoresearch/discussions (accessed 26 March 2026). The \u003cem\u003eautoresearch\u003c/em\u003e codebase is available at https://github.com/karpathy/autoresearch under the MIT License. All other empirical evidence is drawn from publicly available peer-reviewed papers and preprints as cited.\u003c/p\u003e\n\u003cp\u003eEthics Approval\u003c/p\u003e\n\u003cp\u003eNot applicable. This study involves no human participants, no animal subjects, and no personal data.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eB\u0026auml;ck THW, Kononova AV, van Stein B, Wang H, Antonov KA, Kalkreuth RT, de Nobel JP, Vermetten D, de Winter R, Ye F (2023) Evolutionary algorithms for parameter optimization\u0026mdash;thirty years later. Evol Comput 31(2):81\u0026ndash;122. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1162/evco_a_00325\u003c/span\u003e\u003cspan address=\"10.1162/evco_a_00325\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBaratchi M, Wang C, Limmer S, van Rijn JN, Hoos H, B\u0026auml;ck T, Olhofer M (2024) A literature review on automated machine learning. Artif Intell Rev. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10462-024-10726-1\u003c/span\u003e\u003cspan address=\"10.1007/s10462-024-10726-1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBengio Y, Louradour J, Collobert R, Weston J (2009) Curriculum learning. In: Proceedings of the 26th annual international conference on machine learning (ICML 2009). ACM, New York, pp 41\u0026ndash;48. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/1553374.1553380\u003c/span\u003e\u003cspan address=\"10.1145/1553374.1553380\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChan JS, Chowdhury N, Jaffe O, Aung J, Sherburn D, Mays E, Starace G, Liu K, Maksin L, Patwardhan T, Weng L, Mądry A (2024) MLE-bench: evaluating machine learning agents on machine learning engineering. arXiv:2410.07095\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChi Y, Lin Y, Hong S, Pan D, Fei Y, Mei G, Liu B, Pang T, Kwok J, Zhang C (2024) SELA: tree-search enhanced LLM agents for automated machine learning. arXiv:2410.17238\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGardner J, Dutta A, B\u0026ouml;l\u0026ouml;ni L (2026) From language to action: a review of large language models as autonomous agents and tool users. Artif Intell Rev. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10462-025-11471-9\u003c/span\u003e\u003cspan address=\"10.1007/s10462-025-11471-9\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGuo J, Li Z, Ma D, Zheng T, Zang Q, Liu M, Niu X, Wang Y, Yang J, Liu J (2024) DS-Agent: automated data science by empowering large language models with case-based reasoning. arXiv:2402.17453\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang Q, Vora J, Liang P, Leskovec J (2024) MLAgentBench: evaluating language agents on machine learning experimentation. In: Proceedings of the 41st international conference on machine learning (ICML 2024). PMLR, pp 20265\u0026ndash;20289. arXiv:2310.03302\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJaderberg M, Dalibard V, Osindero S, Czarnecki WM, Donahue J, Razavi A, Vinyals O, Green T, Dunning I, Simonyan K (2017) Population based training of neural networks. arXiv:1711.09846 [preprint]\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJiang Z, Schmidt D, Srikanth D, Xu D, Kaplan I, Jacenko D, Wu Y (2025) AIDE: AI-driven exploration in the space of code. arXiv:2502.13138\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKarpathy A (2026) autoresearch: AI agents running research on single-GPU nanochat training automatically. GitHub repository. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/karpathy/autoresearch\u003c/span\u003e\u003cspan address=\"https://github.com/karpathy/autoresearch\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Accessed 26 Mar 2026\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007-001. Keele University, Keele\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi W, Li Z, Li Y (2025) A survey of automatic prompt engineering: an optimization perspective. arXiv:2502.11560\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi Z, Zang Q, Ma D, Guo J, Zheng T, Liu M, Niu X, Wang Y, Yang J, Liu J, Zhong W, Zhou W, Huang W, Zhang G (2024b) AutoKaggle: a multi-agent framework for autonomous data science competitions. arXiv:2410.20424\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu S, Chen J, Ye S, Li Q, Li J, Yu J, Liu Y, Li H, Sun Y, Ying R (2024a) AgentHPO: large language model agent for hyper-parameter optimization. arXiv:2402.01881\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLu C, Lu C, Lange RT, Foerster J, Clune J, Ha D (2024) The AI Scientist: towards fully automated open-ended scientific discovery. arXiv:2408.06292\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLupidi A, Gauri B, Foster TS, Al Omari B, Magka D, Pepe A, Audran-Reiss A et al (2026) AIRS-Bench: a suite of tasks for frontier AI research science agents. arXiv:2602.06855\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMcKinney E (2026) Andrej Karpathy's new open source 'autoresearch' lets you run hundreds of AI experiments a night. VentureBeat, 10 Mar 2026. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://venturebeat.com/technology/andrej-karpathys-new-open-source-autoresearch-lets-you-run-hundreds-of-ai\u003c/span\u003e\u003cspan address=\"https://venturebeat.com/technology/andrej-karpathys-new-open-source-autoresearch-lets-you-run-hundreds-of-ai\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Accessed 26 Mar 2026 [media report; not peer-reviewed]\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNathani D, Madaan L, Roberts N, Bashlykov N, Menon A, Moens V, Budhiraja A, Magka D, Vorotilov V, Chaurasia G, Hupkes D, Cabral RS, Shavrina T, Foerster J, Bachrach Y, Wang WY, Raileanu R (2025) MLGym: a new framework and benchmark for advancing AI research agents. arXiv:2502.14499\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNovikov A, Vu N, Eisenberger M, Dupont E, Huang P-S, Wagner AZ, Shirobokov S, Kozlovskii B, Ruiz FJR, Mehrabian A (2025) AlphaEvolve: a coding agent for scientific and algorithmic discovery. arXiv:2506.13131\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePetersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic mapping studies in software engineering. In: Proceedings of the 12th international conference on evaluation and assessment in software engineering (EASE 2008). BCS Learning \u0026amp; Development, Swindon, pp 68\u0026ndash;77\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePlaat A et al (2025) Agentic AI: a comprehensive survey of architectures, applications, and future directions. Artif Intell Rev. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10462-025-11422-4\u003c/span\u003e\u003cspan address=\"10.1007/s10462-025-11422-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRenduchintala HSVNSK, Panwar P, Singh P, Khandelwal A (2024) POSIX: a prompt sensitivity index for large language models. arXiv:2410.02185\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRomera-Paredes B, Barekatain M, Novikov A, Balog M, Kumar MP, Dupont E, Ruiz FJR, Ellenberg JS, Wang P, Fawzi O (2023) Mathematical discoveries from program search with large language models. Nature 624:545\u0026ndash;552. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41586-023-06924-6\u003c/span\u003e\u003cspan address=\"10.1038/s41586-023-06924-6\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSclar M, Choi Y, Tsvetkov Y, Suhr A (2024) Quantifying language models' sensitivity to spurious features in prompt design or: how I learned to start worrying about prompt formatting. In: Proceedings of ICLR 2024. arXiv:2310.11324\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSchmid P (2026) How autoresearch will change small language models adoption. philschmid.de. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.philschmid.de/autoresearch\u003c/span\u003e\u003cspan address=\"https://www.philschmid.de/autoresearch\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Accessed 26 Mar 2026 [secondary report; claims require independent replication]\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSchmidgall S, Su Y, Wang Z, Sun X, Wu J, Yu X, Liu J, Liu Z, Barsoum E (2025) Agent Laboratory: using LLM agents as research assistants. arXiv:2501.04227\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSnoek J, Larochelle H, Adams RP (2012) Practical Bayesian optimization of machine learning algorithms. Advances in neural information processing systems, vol 25. Curran Associates, Red Hook, pp 2951\u0026ndash;2959\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTang J, Xia L, Li Z, Huang C (2025) AI-Researcher: autonomous scientific innovation. In: Advances in neural information processing systems. Curran Associates, Red Hook. arXiv:2505.18705\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eToledo E, Hambardzumyan K, Josifoski M, Hazra R, Baldwin N, Audran-Reiss A, Kuchnik M, Magka D, Jiang M, Lupidi AM et al (2025) AI research agents for machine learning: search, exploration, and generalisation in MLE-bench. arXiv:2507.02554 (v2, November 2025)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003evan Stein N, B\u0026auml;ck THW (2024) LLaMEA: a large language model evolutionary algorithm for automatically generating metaheuristics. IEEE Trans Evol Comput. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/TEVC.2024.3511703\u003c/span\u003e\u003cspan address=\"10.1109/TEVC.2024.3511703\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003evan Stein N, Vermetten DL, B\u0026auml;ck THW (2026) In-the-loop hyper-parameter optimization for LLM-based automated design of heuristics. ACM Trans Evol Learn Optim. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3729220\u003c/span\u003e\u003cspan address=\"10.1145/3729220\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang L, Ma C, Feng X, Zhang Z, Yang H, Zhang J, Chen Z, Tang J, Chen X, Lin Y, Zhao WX, Wei Z, Wen JR (2024) A survey on large language model based autonomous agents. Front Comput Sci 18(6):186345. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s11704-024-40231-1\u003c/span\u003e\u003cspan address=\"10.1007/s11704-024-40231-1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang Z, Wang K, Wang Q, Zhang P, Li L, Yang Z, Jin X, Yu K, Nguyen MN, Liu L, Gottlieb E, Lu Y, Cho K, Wu J, Fei-Fei L, Wang L, Choi Y, Li M (2025) RAGEN: understanding self-evolution in LLM agents via multi-turn reinforcement learning. arXiv:2504.20073\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWei J, Wang X, Schuurmans D, Bosma M, Xia F, Chi E, Le QV, Zhou D (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, vol 35. Curran Associates, Red Hook, pp 24824\u0026ndash;24837\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWistuba M, Schilling N, Schmidt-Thieme L (2015) Hyperparameter search space pruning\u0026mdash;a new component for sequential model-based hyperparameter optimization. In: Proceedings of ECML PKDD 2015. Springer, Berlin, pp 104\u0026ndash;119\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYamada Y, Lange RT, Lu C, Hu S, Lu C, Foerster J, Clune J, Ha D (2025) The AI Scientist-v2: workshop-level automated scientific discovery via agentic tree search. arXiv:2504.08066\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang X, Yang X, Fang S, Xian B, Li Y, Wang J, Xu M, Pan H, Hong X, Liu W, Shen Y, Chen W, Bian J et al (2025) R\u0026amp;D-Agent: an LLM-agent framework towards autonomous data science. arXiv:2505.14738 (v2, October 2025)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZheng Y, Zhang L, Li K, Wang R, Li W, Zhang T, Zhang Q et al (2026) A survey on large language models driven meta-optimizers for automated intelligent optimization. Artif Intell Rev 59:72. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10462-025-11470-w\u003c/span\u003e\u003cspan address=\"10.1007/s10462-025-11470-w\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhuo J, Zhang S, Fang X, Duan H, Lin D, Chen K (2024) ProSA: assessing and understanding the prompt sensitivity of LLMs. Findings of ACL: EMNLP 2024. ACL, Miami, pp 1950\u0026ndash;1976\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"artificial-intelligence-review","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"aire","sideBox":"Learn more about [Artificial Intelligence Review](http://link.springer.com/journal/10462)","snPcode":"10462","submissionUrl":"https://submission.nature.com/new-submission/10462/3","title":"Artificial Intelligence Review","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"autonomous research agents, agentic AutoML, research program design, instruction engineering, LLM-guided optimisation","lastPublishedDoi":"10.21203/rs.3.rs-9286871/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9286871/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAutonomous machine learning experimentation systems\u0026mdash;wherein a large language model (LLM) agent iteratively proposes, executes, and evaluates code modifications against a fixed scalar metric\u0026mdash;represent a fundamental shift in how machine learning research is conducted. In these systems, the practitioner's primary lever is not the training code itself but the natural-language \u003cem\u003eresearch program\u003c/em\u003e: the instruction document that specifies objectives, priorities, and constraints for the agent across dozens or hundreds of consecutive decisions. Despite this centrality, no principled framework for designing research programs exists in the literature. This survey addresses that gap through four contributions. First, we conduct a structured cross-system analysis of sixteen agentic AutoML and autonomous research systems\u0026mdash;including AIDE, AIRA, R\u0026amp;D-Agent, AgentHPO, AlphaEvolve, MLAgentBench, AI-Researcher, and AI Scientist-v2\u0026mdash;identifying the instruction document as a universal practitioner-facing control mechanism and cataloguing seven design dimensions. Second, we develop a five-family taxonomy of instruction strategies: Scope-Constrained, Hypothesis-Directed, Diversity-Preserving, Simplicity-Biased, and Curriculum-Staged, grounded in theory from the AutoML, evolutionary computation, prompt engineering, and curriculum learning literatures. Third, we provide multi-source empirical grounding: analysis of two publicly documented overnight sessions suggests a cross-session curriculum intervention is associated with a 37% difference in total gain, with important caveats regarding session-length confounding; independently controlled benchmarks from AIRA and AgentHPO corroborate the taxonomy's predictions. Fourth, five practitioner guidelines with explicitly labelled calibration thresholds are synthesised and validated against all sixteen surveyed systems.\u003c/p\u003e","manuscriptTitle":"Instruction Strategy Design for Autonomous Machine Learning Experimentation Systems: A Taxonomy, Cross-System Analysis, and Evidence-Based Practitioner Framework","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-02 08:03:02","doi":"10.21203/rs.3.rs-9286871/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorInvitedReview","content":"","date":"2026-05-10T15:06:45+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-06T07:14:10+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-04T14:31:18+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-30T10:58:27+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"20849621020857259607635948268725693533","date":"2026-04-22T16:06:40+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"253865245349413169147858863705070714738","date":"2026-04-21T09:50:27+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"79773656981768059140543504887320886290","date":"2026-04-20T11:32:05+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"310227386101867641475115622138953042063","date":"2026-04-20T03:59:31+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"112507648738080357129759216417321682867","date":"2026-04-20T03:57:38+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-20T02:30:29+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-04-07T03:05:44+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-04-02T05:05:30+00:00","index":"","fulltext":""},{"type":"submitted","content":"Artificial Intelligence Review","date":"2026-04-01T05:09:28+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"artificial-intelligence-review","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"aire","sideBox":"Learn more about [Artificial Intelligence Review](http://link.springer.com/journal/10462)","snPcode":"10462","submissionUrl":"https://submission.nature.com/new-submission/10462/3","title":"Artificial Intelligence Review","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"051f807c-b966-4b6a-aca4-301f3cccbe0e","owner":[],"postedDate":"April 2nd, 2026","published":true,"recentEditorialEvents":[{"type":"editorInvitedReview","content":"","date":"2026-05-10T15:06:45+00:00","index":26,"fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-06T07:14:10+00:00","index":25,"fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-04T14:31:18+00:00","index":24,"fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-30T10:58:27+00:00","index":23,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-04-20T02:38:23+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-02 08:03:02","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9286871","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9286871","identity":"rs-9286871","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}