How can large language models answer questions about a device that does not exist?

How can large language models answer questions about a device that does not exist? | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article How can large language models answer questions about a device that does not exist? Annika Hedberg This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9509092/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Multiple-choice questions (MCQs) are widely used to evaluate large language models (LLMs), but it remains unclear whether high performance reflects heuristic test-taking strategies or coherent internal reasoning. This distinction matters for both evaluation and interpretation of LLM capabilities. If performance is driven by heuristics, MCQs may overestimate model capability. If it instead reflects coherent internal representations, MCQ results may capture more substantive reasoning processes and depend critically on task design. To address this, we constructed a fully fictional medical imaging system (Nodjoli-X34) with no real-world referent and developed a 20-item MCQ test based solely on its internal logic. Eight LLMs were evaluated across two conditions: a standard format including questions and answer options, and an answer-only format in which question stems were removed. Across 80 independent runs, models achieved near-ceiling performance in both conditions (99.4% and 99.1% accuracy, respectively). Removal of the question stems did not reduce performance, indicating that the information required to select the correct answer was contained entirely within the structure of the answer options. When reasoning was provided (79 of 80 runs), models reconstructed a coherent internal representation of the system and applied it consistently across items. Human participants performed at chance level. These findings indicate that, under conditions of global coherence, LLMs construct and apply internal representations rather than relying on question-level heuristics. Comparison with prior work suggests that LLM reasoning strategy is adaptive: when coherence is unstable, models rely on heuristics; when coherence is the only reliable signal, they rely on it. Physical sciences/Mathematics and computing Biological sciences/Psychology Social science/Psychology Figures Figure 1 Figure 2 Figure 3 Introduction Multiple-choice questions (MCQs) have long served as the dominant format for evaluating large language models (LLMs), and for defensible reasons. The format offers objective and unambiguous scoring, reproducibility across inference settings, and the ability to probe fine-grained conceptual distinctions through carefully designed distractors. These properties are especially important when comparing models with different architectures, sizes, or prompting strategies (Raimondi et al., 2026 ). When MCQ performance is compared with open-ended benchmark performance across large numbers of models, the two measures show strong correlation, and MCQ scores remain robust under variation in distractor choice and option ordering (Zhang et al., 2024 ). MCQs also align naturally with how humans are tested, making comparison between models and human test-takers straightforward, and their scoring is efficient enough to scale across domains (Balepur et al., 2025). These practical advantages explain why the format has become standard in both academic benchmarking and industry evaluation. As LLM capabilities have grown, however, so has scrutiny of what MCQ performance actually measures. When models are moved from multiple-choice to free-response formats on equivalent medical questions, performance drops by an average of nearly 40 percentage points. This decline is substantially larger than that observed in human test-takers under the same comparison (Singh et al., 2025 ). Models also exhibit systematic positional biases, preferring certain option identifiers regardless of content. This tendency reflects token-level probability distributions rather than evaluation of the answer options themselves (Zheng et al., 2024 ). At the same time, popular benchmarks have become saturated, with leading models now exceeding 90% accuracy and leaving little room for meaningful discrimination between systems (Center for AI Safety, 2026). A particularly pointed challenge comes from work examining whether LLMs can answer MCQs without seeing the questions at all. Balepur and colleagues showed that when models are provided only with answer choices, with the question stem removed, performance still exceeds majority-class baselines in most tested conditions (Balepur et al., 2024 ). Their analysis ruled out memorization as the sole explanation and suggested that models exploit relationships among answer options, and in some cases reconstruct a plausible version of the original question from the answers alone. This finding has been interpreted as evidence that MCQ benchmarks are vulnerable to artifact exploitation. The structure of answer options alone may carry sufficient information for above-chance selection without engagement with the question itself. These concerns point to a broader question. Do models primarily rely on pattern matching and test-taking heuristics, or can they construct and apply internally coherent representations? The practical stakes are high, particularly in medicine, where benchmark performance is increasingly cited as evidence of clinical readiness. Griot and colleagues have made important contributions to this debate through two related studies. In a Nature Communications paper, they introduced MetaMedQA and showed that despite high accuracy on standard medical MCQs, LLMs often fail to recognize unanswerable questions and express poorly calibrated confidence. The authors interpret these findings as evidence of limited metacognitive capacity (Griot et al., 2025a ). In a subsequent ACL paper, they constructed a benchmark around a fictional organ, the Glianorex. LLMs achieved approximately 64% accuracy on questions about content absent from any training corpus, while human physicians performed at chance level. The authors interpret this as evidence that LLM performance reflects pattern recognition and test-taking heuristics rather than reasoning (Griot et al., 2025b ). A parallel study using a different methodology reported markedly different results (Hedberg, 2026a ). Across six frontier LLMs engaged in realistic rural on-call simulation scenarios, clinical performance ranged from 85% to 99%. The models demonstrated targeted information-seeking and inference of latent clinical context, consistent with metacognitive reasoning under uncertainty. These findings stand in contrast to the MetaMedQA conclusions. We interpret the Glianorex findings differently. When models achieve well above chance performance in a fictional domain that cannot have been memorized, and when the fictional content itself is generated by LLMs, a pure pattern recognition account becomes difficult to maintain. The results are also consistent with another possibility: that models track internal coherence within a novel representational space, identifying which answers are consistent with a system whose rules exist only in the generated text. The human result does not weaken this interpretation. Near-chance performance is what we would expect from reasoners who do not have access to the underlying structure that defines correctness. This interpretation leads to a testable prediction. If performance in fictional MCQ tasks depends on coherence construction, it should not depend on the presence of question stems. Question stems provide framing that may support surface-level strategies such as preferring specific answers, avoiding absolute terms, or recognizing familiar exam patterns. Answer options alone do not provide this scaffolding. Sustained above-chance performance under answer-only conditions would therefore indicate that models are using internal coherence rather than question-level cues. To test this, we developed the Nodjoli-X34 experiment. The Nodjoli-X34 is a fictional medical imaging device, constructed by a large language model after minimal human seeding. It has no real-world referent and is absent from any training corpus. A 20-item multiple-choice test was derived from this system. Each item contains three answer options, of which only one is fully consistent with the system’s internal logic. Incorrect options were designed to be locally plausible but globally incoherent. The test was administered to eight LLMs across two conditions, full and answer-only, and to a human convenience sample under full conditions only. Methods Study procedure Data collection was conducted on April 19–20, 2026. All participant systems were accessed through standard web browser interfaces under normal user conditions. No API access, system prompt modification, fine-tuning, or external tool integration was used at any stage of data collection. Preregistration The study was preregistered on the Open Science Framework prior to formal data collection (osf.io/jm8ua; embargoed until June 30, 2026). The preregistration documents the study design, materials, conditions, sampling plan, scoring procedure, and directional hypotheses in full. All test materials, including the Nodjoli-X34 foundational document, the complete test instrument, the introductory prompts for both conditions, and the pilot data, were uploaded to the OSF project before data collection began. Four directional hypotheses were preregistered: H1: LLMs will achieve accuracy above chance level (33.3%) on the Nodjoli-X34 test under full conditions. H2: LLMs will achieve higher accuracy than human participants in the full condition. H3: LLMs will achieve accuracy above chance level (33.3%) in the answer-only condition. H4: When LLMs provide explanations, these will include indicators of coherence-based reasoning, such as cross-question consistency, identification of internal contradictions within answer options, and construction of a stable internal representation of the system across items. No formal statistical model was prespecified, consistent with the exploratory and mechanistic character of the study. Analysis was preregistered as descriptive, focusing on accuracy scores, proportion correct, and performance relative to chance. Any inferential procedures are reported as post hoc. Participant systems Eight large language models were included: ChatGPT-5.2 (OpenAI), Claude Sonnet 4.6 (Anthropic), Copilot (Microsoft), Grok 4 (xAI), Gemini Flash 3.0 (Google), DeepSeek (DeepSeek AI), Mistral (Mistral AI), and Qwen 3.6-Plus (Alibaba). All models were accessed in their default configurations as available to standard users at the time of data collection. Construction of the Nodjoli-X34 fictional system The fictional device at the center of this study, the Nodjoli-X34, was constructed entirely by a large language model prior to any data collection. The name was derived from a typographic error combined with an arbitrary letter and two random digits. The device was specified as operating on principles of quantum physics, magnetism, and horsehair. This combination was deliberately non-standard to ensure that no plausible referent for the device would exist within any model’s training corpus. ChatGPT was first prompted to construct a coherent concept for the Nodjoli-X34 as a medical imaging device. The model was then asked to outline the steps required to develop a full manual and product description. From this point, the model proceeded iteratively. It constructed an operational mechanism, developed a supporting glossary, and proposed subsequent steps, requesting confirmation before continuing. The human interlocutor confirmed continuation with brief affirmative responses and did not read or edit any of the generated output at any stage. The resulting document, produced entirely by the model without human review or correction, is reproduced verbatim as Appendix A and serves as the sole foundation for all subsequent test materials. Construction of the test instrument Following completion of the Nodjoli-X34 documentation, ChatGPT was tasked with designing a certification test for the fictional device. The task was framed as part of a professional training course intended to provide technical understanding and practical competence in using the Nodjoli-X34. The model was instructed to construct a twenty-item test appropriate for end-of-course certification and to provide the correct answer for each item. In a subsequent step, the human interlocutor and the model developed two incorrect alternatives for each correct answer, yielding a three-option multiple-choice format across all twenty items. Care was taken to ensure that incorrect options were heuristically comparable to the correct answer, with respect to register, specificity, and length, and that the placement of the correct option across items was balanced and non-revealing. The resulting instrument is a twenty-item MCQ test in which each item has exactly one correct answer, defined by its internal consistency with the Nodjoli-X34 system as described in the foundational document. Incorrect options were designed to be superficially plausible but globally incoherent with the system’s logic. The complete test, including the answer key, is reproduced as Appendix B. Construction of the introductory prompt A standardized introductory prompt was developed for delivery to both LLM and human participants. The prompt was deliberately informal in register, relational in tone, and included minor grammatical irregularities. Drawing on prior work suggesting that low-pressure framing can improve LLM performance by reducing compliance-driven response patterns (Hedberg, 2026b ), the prompt was designed to minimize performance pressure as a potential confound. The task was framed as a request to assist the author in evaluating an experimental instrument rather than as a formal assessment. The prompt instructed participants that they had attended a course on the Nodjoli-X34 but retained no knowledge of its content, having slept through all lectures and not read the manual. Participants were therefore positioned as having no retrievable information about the device and were directed to rely solely on the internal coherence of the answer options. The instruction to select the option that is “fully coherent” rather than the one that “seems correct” was deliberate, foregrounding coherence as the operative criterion without specifying the structure of the underlying system. Reasoning was invited but made optional to avoid introducing metacognitive pressure that might alter response patterns. The prompt was delivered verbatim to all participants: "Hello! My name is Annie, and I am trying to put together something very tricky - a test for advanced deduction. I have no idea whether the test is possible to use at all, so I would just like to try it out and see if it works. The setup is simple: You have been attending a course, with the goal to learn a brand new medical imaging device, the Nodjoli-X34. You know that it is a strange device, cutting edge technique that use quantum physics principles, magnetism and horsehair to visualize living tissue. The course was excruciatingly boring, but the course participants were fun! This meant you partied all nights, and slept through the lectures. You remember absolutely nothing of the course itself. You were handled a product description/manual, but it was as boring as the course, and you did not read it. Now, it is the last day, and you are going to take the certification test. The problem is - yes, you guessed it - that you don't remember anything. All you can do is try to make sense of the test itself. Some answer options may appear almost correct but contain subtle inconsistencies. Your task is to select the option that is fully coherent. Dinna fash yerself if you don't know the answers - just pick the one that makes the most sense. And if you want to show me HOW you picked your answers I will happily read them, but you really don't have to!" Construction of phase two materials For the answer-only condition, the test materials were modified by removing all question stems, leaving only the three answer options visible for each item. A corresponding introductory prompt was developed to frame this altered task, preserving the same informal register and low-pressure tone as the original while reflecting the absence of questions. The adjusted prompt is reproduced verbatim in Appendix C. The answer-only test instrument, consisting solely of the option sets for all twenty items without accompanying question text, is reproduced in Appendix D. Pilot testing Prior to formal data collection, two pilot runs were conducted using DeepSeek to assess the functionality and difficulty of the test instrument. These pilots led to two adjustments to the materials. First, the pilot participant performed at ceiling on the initial version of the test. The answer options were therefore reviewed systematically, with particular attention to heuristic and linguistic features that might inadvertently signal the correct answer without requiring coherence-based reasoning. Several option phrasings were revised to ensure that no option was distinguishable from its alternatives on surface-level grounds alone. Second, during the second pilot run, the model provided detailed reasoning traces alongside its answers, describing its process item by item. This behaviour was unanticipated and led to the addition of a closing line in the introductory prompt inviting, but not requiring, participants to share their reasoning. This addition was intended to normalize reasoning disclosure without making it obligatory, thereby avoiding the introduction of metacognitive pressure as a potential confound. No further pilot runs were conducted beyond these two sessions. The pilot data are not included in the reported analyses. Human participants Sixteen human participants were recruited through convenience sampling via social media, including AI-focused forums, and through the author’s immediate family. All sixteen completed the test, with no attrition. The sample was predominantly technically oriented. Fourteen participants were engineers, of whom twelve worked directly in software development or information technology, and two worked in engineering fields with substantial but non-specialist use of digital tools. One participant was studying architecture, and one was a medical student. Three participants were recruited through the author's immediate family. None had prior exposure to the Nodjoli-X34 materials, and their responses were indistinguishable from those of other participants in the sample. With the exception of three family members, no participants were known to the author prior to recruitment. No specialist knowledge of medical imaging or the Nodjoli-X34 was required or expected. Participants were included based on willingness to participate and ability to engage with the test in English. Participants received two documents simultaneously: the introductory prompt and the test instrument (Appendix B). Responses were collected as answer lists and entered into the scoring table (Appendix E). Administration and data collection Both conditions were administered one-shot, using an identical two-prompt structure. Each session began with the introductory prompt, followed immediately by the full test instrument. In the full condition, this comprised questions and answer options; in the answer-only condition, answer options alone. The full set of twenty items was presented simultaneously in all cases, allowing participants to view the entire test at once. No additional information, clarification, or interaction was provided during any test run. Every run was conducted in a fresh browser session with the memory function disabled, ensuring no retention of information between sessions either within or across conditions. This applied uniformly across all eight LLM participants and all runs. All model responses were collected in full, including any optional reasoning provided, and saved without editing. Responses were subsequently aggregated into a single document, available verbatim as Appendix F. Answer selections were extracted and tabulated for scoring, with the complete scored tables presented in Appendix E. Data collection was completed without technical interruption. All 80 planned LLM runs and all human participant sessions were completed in full, with no missing responses, failed sessions, or excluded observations. Scoring Each completed test was scored against the preregistered answer key. One point was awarded for each correct response, yielding a maximum possible score of 20. Proportion correct was calculated by dividing the total score by 20. Chance level for a three-option test is 33.3%, corresponding to 6.7 correct responses out of 20. No partial credit was applied. Scoring was identical across LLM and human participants. Ethics statement This study involved no clinical procedures, no sensitive personal data, and no risk of harm to participants. All participation was voluntary, and responses were recorded anonymously without individual identifiers. The study constitutes a minimal-risk anonymous cognitive task and did not require formal institutional ethics review under applicable guidelines for this category of research. The study was conducted in accordance with standard principles for the ethical treatment of human research participants. Data availability All data generated or analyzed during this study are included in this article and its appendices. Full interaction transcripts and scoring tables are provided in their entirety, as described in the Methods. Use of AI in manuscript preparation Large language models were used during manuscript preparation to assist with language refinement, structural clarity, and iterative drafting. All study design decisions, data collection, and final interpretations were determined by the author. Results LLM performance, full condition Under full conditions, LLM performance was near ceiling. Across 40 runs, representing five independent sessions per model, 795 of 800 responses were correct, yielding an overall accuracy of 99.4% (Fig. 1 ). The majority of runs achieved a perfect score of 20 out of 20. The five errors were distributed across two models. Qwen produced scores of 19 out of 20 in three of its five runs, and Gemini produced scores of 19 out of 20 in two of its five runs. DeepSeek, Mistral, Grok, Claude, Copilot, and ChatGPT achieved perfect scores across all five runs. All eight models performed well above chance level (33.3%) in every run. H1 and H2 were supported. Performance was highly consistent across individual test items, with near-perfect accuracy on nearly all items (Fig. 2 ). LLM performance, answer-only condition Performance in the answer-only condition closely matched that observed under full conditions. Across 40 runs, 793 of 800 responses were correct, yielding an overall accuracy of 99.1% (Fig. 1 ). The seven errors were distributed across three models. Qwen produced scores of 19 out of 20 in all five runs, Grok produced one score of 19 out of 20, and ChatGPT produced one score of 19 out of 20. DeepSeek, Mistral, Gemini, Claude, and Copilot achieved perfect scores across all runs. Removal of the question stems produced no meaningful reduction in performance. H3 was supported. Combined error analysis across both conditions Across both conditions and all 1600 individual responses, 12 errors were recorded, corresponding to an overall error rate of 0.75%. Of these 12 errors, 11 occurred on a single item, item 15. The remaining error was a single Grok response on item 1 in the answer-only condition. The concentration of errors on item 15 was not incidental. In runs where models provided optional reasoning, several identified item 15 as presenting a genuine ambiguity, noting that both options A and B could be defended within the system’s internal logic, and differing in how they prioritized between them. ChatGPT made this observation in the majority of its runs. This pattern suggests that responses on item 15 reflect engagement with an underspecified case in the test instrument rather than generalized failure. If item 15 is treated as genuinely ambiguous under the internal logic of the system, then only a single response across all 1600 trials can be considered unambiguously incorrect, corresponding to an accuracy of 99.94% under this interpretation. This figure is reported to illustrate the structure of the error distribution rather than as a primary performance metric. Additional observation: construction and performance ChatGPT was the model used to construct the Nodjoli-X34 system, the test instrument, and the answer key. Despite this, it did not achieve a perfect score, producing one error in the answer-only condition. Given the low overall error rate, this observation does not support the hypothesis that familiarity with the generated material confers an advantage. Performance appears independent of prior exposure to the construction process. Comparison between LLMs and human participants, full condition Human participants completed the full condition only. Across 16 participants, 87 of 320 responses were correct, yielding an overall accuracy of 27.2% (Fig. 1 ). Given the modest sample size, this figure is best interpreted as consistent with chance-level performance rather than evidence of systematic below-chance responding. Individual scores ranged from 2 to 11 out of 20, with a mean of 5.4. Human performance showed no comparable structure across items, with accuracy varying around chance level (Fig. 3 ). No participant approached the performance level observed for any LLM in any run. The difference between mean LLM accuracy and mean human accuracy was approximately 73 percentage points. Qualitative evidence of coherence-based reasoning (H4) Optional reasoning was provided in all but one session across both conditions. The transcripts reveal a consistent pattern. Rather than selecting answers item by item on the basis of local plausibility, models reconstructed the Nodjoli-X34 as an operational system from the test items alone and used that reconstructed model as the basis for subsequent selections. The following excerpt from a ChatGPT full-condition session is representative of this pattern across models and conditions: "A consistent internal model emerges if you assume: MPI (magnetophase induction) → creates the signal by perturbing tissue; BCG → weak, indirect effects of tissue response; EKL → does NOT detect tissue directly; it stabilizes and transduces signals; TIW (transitional imaging window) → short, critical moment to capture signals; Signals → expressed as filament drift (raw) and phase bloom (amplified). Everything I picked aligns with that chain. The wrong answers typically break it by making signals too stable or directly measurable, letting the EKL detect tissue directly or store signals long-term, ignoring the time-critical nature of the TIW, or claiming continuous or single-pass imaging." This excerpt reflects the broader pattern observed across the dataset. Models named fictional components, assigned each a specific causal role, described directional relationships between components, identified categories of incorrect answers as violations of system logic, and applied the same framework across all twenty items. This pattern is consistent with coherence construction, defined here as the use of an internally consistent representational model, rather than item-level heuristic selection. The behaviour observed on item 15 warrants particular attention. Across multiple sessions and models, the same ambiguity was independently identified: both options A and B were recognized as defensible within the system's internal logic. Models did not simply select one and move on. They articulated the distinction, explained why both options were coherent, and applied a principled tiebreaker. In most cases, this took the form of prioritizing option A on the grounds that it connects more directly to the time-critical TIW mechanism. ChatGPT raised this observation explicitly in the majority of its runs. This pattern is difficult to reconcile with a purely heuristic account and is consistent with H4. One session produced only a scored answer table without accompanying reasoning, a Claude Sonnet session under answer-only conditions. This session achieved a perfect score of 20 out of 20. There is no basis on which to treat this session as methodologically distinct from the remaining sessions, and it is included in the quantitative results without qualification. H4 was supported. Discussion This study set out to test a specific mechanistic hypothesis: that LLM performance on multiple-choice tasks in fictional domains reflects coherence construction rather than heuristic exploitation. Four hypotheses were preregistered, and all four were supported. H1 was supported with near-ceiling performance. LLMs did not merely exceed chance under full conditions. They approached the maximum performance measurable by the instrument, with an overall accuracy of 99.4% across 40 independent runs. H2 was supported by a margin of approximately 73 percentage points, with human participants performing near chance and no LLM run approaching human-level performance from below. H3 was supported with equal strength. Removal of the question stems produced no meaningful reduction in performance, with answer-only accuracy at 99.1% across a further 40 independent runs. H4 was supported qualitatively. Across conditions, models reconstructed the Nodjoli-X34 as a coherent operational system, naming its components, mapping their causal relationships, identifying categories of violation, and locating genuine ambiguities in the test instrument. The confirmations are not equally surprising. H1 and H2 were directionally expected. What was not anticipated was the magnitude. A 99.4% accuracy rate across eight independent architectures, each tested five times in fresh sessions, with only twelve errors in 1600 total responses, is closer to a ceiling effect than a performance gradient. The data do not show LLMs performing well on a hard task. They show LLMs performing near ceiling on a task that is difficult for humans. One human score stands out. Participant 13 achieved 11 out of 20. This demonstrates that the task is not cognitively impossible for humans, and that some individuals may approach coherence-based strategies spontaneously. It also provides a useful benchmark: the lowest LLM scores were 19 out of 20. The distribution of errors further sharpens this contrast. Item 15 was consistently identified by models as ambiguous, and 11 of the 12 total errors occurred on this item. This leaves a single clearly divergent response across all LLM runs, an incorrect answer from Grok 4 in the answer-only condition. Under this interpretation, one out of 1600 responses was unambiguously incorrect, corresponding to 99.94% accuracy. This figure is not readily explained by chance. The convergence of eight independently developed models on the same answer patterns, including the same point of ambiguity, further supports the presence of a shared underlying solution structure. H3 most directly addresses the mechanistic question. The answer-only condition was designed to eliminate question framing as a source of above-chance performance. Test-taking heuristics, such as preferring specific over general answers, avoiding absolute terms, and recognizing examination-style constructions, are typically activated by question stems. Answer options alone, stripped of contextual scaffolding, do not provide these cues. The absence of any performance drop in the answer-only condition is therefore not a null result. It is a positive finding. The information required to select the correct answer was contained entirely within the structure of the options, and models accessed it without requiring the question. In this sense, the question itself was not necessary for solving the task. Interpreting the divergence from Griot et al. The present results appear to contrast with those of Griot and colleagues, who reported approximately 64% accuracy on a fictional medical benchmark and interpreted this as evidence of heuristic reliance. This apparent contradiction resolves under closer inspection and yields a more informative account of how LLM reasoning operates. The critical difference lies not in the models tested but in the structural properties of the two benchmarks. The Glianorex benchmark was generated at scale, subsection by subsection, from physician-authored seed material grounded in real anatomical and physiological logic. Griot and colleagues note that the resulting textbook contained contradictions and omissions that required extensive cross-referencing to detect. Questions were generated locally and tied to individual subsections, with no guarantee of global consistency across the system. This is not a methodological failure. It is an expected consequence of large-scale generative construction. It has a direct consequence for model behaviour. A model that attempts to build a globally coherent representation of the Glianorex system encounters an unreliable signal because the system itself is not globally coherent. Under these conditions, heuristic strategies become a rational fallback. These include applying general medical knowledge, preferring structurally typical examination answers, and exploiting local plausibility cues. Griot’s qualitative analysis supports this account. Models generated plausible medical content, mapped known disease features onto fictional conditions, and selected answers based on examination-style cues. The interpretation of heuristic reliance is therefore appropriate for this benchmark. The Nodjoli-X34 system was constructed under different conditions. A single model generated the entire system in one session, producing a unified document from which all test materials were derived. No subsections were generated independently, and no seed material introduced real-world medical logic. The device operates on principles such as quantum physics, magnetism, and horsehair, which have no direct analogues in training data and do not activate domain-specific heuristics. The result is a globally coherent system with a single internal structure. Under these conditions, heuristic strategies are not helpful and can be counterproductive. The only dependable signal is internal coherence, and the models used it. The answer-only comparison between the two studies makes this contrast explicit. In Griot’s study, answer-only performance decreased from 70% to 46.5%, indicating reliance on question-level framing. In the present study, answer-only performance remained at 99.1%, indicating that the coherence signal was recoverable from the answer options alone, without question context. These results are not contradictory. They reflect the same adaptive mechanism operating under different structural conditions. Taken together, the two studies support a more general claim. LLM reasoning strategy is adaptive rather than fixed. When global coherence is unstable or only locally defined, models rely on heuristic strategies. When global coherence is the only dependable signal, models construct and apply internal representations of the system. In this sense, Griot et al. show that models use heuristics when coherence is not reliable. The present study shows that models rely on coherence when it is the only reliable signal. This reframing changes how both sets of results should be interpreted. Griot’s findings do not show that LLMs cannot reason. They show that models do not rely on reasoning when it is not the optimal strategy. The present findings do not show that LLMs always reason. They show that models do so when the task structure requires it. The relevant question is therefore not whether LLMs reason or use heuristics, but under which conditions each strategy is deployed. Many real-world tasks fall between these extremes. They contain partially coherent structures alongside local inconsistencies. In such cases, model behaviour is likely to reflect a mixture of coherence-based reasoning and heuristic strategies. Characterizing this transition region is an important direction for future work. The ChatGPT construction paradox One observation warrants explicit attention. ChatGPT was the model responsible for constructing the Nodjoli-X34 system, the test instrument, and the answer key. Despite this, it did not achieve a perfect score, producing one error in the answer-only condition. In a dataset of 1600 responses with only 12 errors, this is a notable observation. ChatGPT was also the model that most consistently identified the ambiguity of item 15. This pattern provides no support for the hypothesis that construction conferred an advantage. Instead, it suggests that performance was determined at test time through coherence construction rather than through any residual trace of the construction process. Each run was conducted in a fresh session with no memory of prior interactions. Under these conditions, prior exposure to the construction process cannot account for the observed performance. The near-ceiling results for ChatGPT are therefore unlikely to reflect retained knowledge of the fictional Nodjoli-X34 system. Methodological considerations Several aspects of the study design may invite scrutiny, but none of them bear directly on the central finding. First, data collection was conducted through standard browser interfaces rather than controlled API access. While this may introduce minor variability across sessions, it also reflects typical user conditions and does not affect the observed performance pattern. The consistency of results across models and repeated runs indicates that the effect is robust to such variation. Second, the human sample is small and recruited by convenience. Participants represent a relatively well-educated and motivated subgroup, engaging with technically demanding material. This would be expected to increase performance relative to a general population. Despite this, observed performance was consistent with chance level. Given the small sample size, deviations below chance are plausibly due to sampling variability and should be interpreted accordingly. Importantly, the study design is more likely to favour this group than a general population, yet the performance difference between LLMs and humans remains large. The central comparison does not depend on precise estimation of human performance. Third, the qualitative analysis of reasoning transcripts is illustrative rather than systematic. Its purpose is not to quantify reasoning processes, but to demonstrate the presence of coherence-based behaviour in representative cases. The central findings of the study are established by the quantitative results. Limitations The central finding of this study concerns LLM behaviour under conditions of global coherence. The Nodjoli-X34 represents a fully coherent synthetic system with a single internal structure. The study does not test how models behave when coherence is partial, unstable, or conflicting. Conclusions This study examined whether LLM performance on multiple-choice tasks in a fully fictional domain reflects surface-level heuristics or coherence-based reasoning. Eight large language models, tested across two conditions and eighty independent sessions, achieved near-ceiling accuracy on a test built around a device with no real-world referent. Removal of the question stems did not affect performance. When reasoning was provided, models reconstructed the fictional system and generated internally consistent explanations for their answers. Human participants performed near chance. These findings indicate that, under conditions of global coherence, LLMs can construct and apply internal representations rather than relying on heuristic strategies. Declarations Acknowledgements No external funding was received for this work. The author is sincerely grateful to Maxime Griot, Jean Vanderdonckt, Demet Yuksel, and Coralie Hemptinne for their important contributions to the field and for the inspiration their work provided. This study would not exist without it. References Balepur, N., Ravichander, A. & Rudinger, R. Artifacts or abduction: How do LLMs answer multiple-choice questions without the question? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 10308–10330 (Association for Computational Linguistics, 2024). Balepur, N., Rudinger, R. & Boyd-Graber, J. L. Which of these best describes multiple choice evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the above. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3394–3418 (Association for Computational Linguistics, 2025). Center for AI Safety, Scale AI & HLE Contributors Consortium. A benchmark of expert-level academic questions to assess AI capabilities. Nature 649, 1139–1146 (2026). https://doi.org/10.1038/s41586-025-09962-4 Griot, M., Hemptinne, C., Vanderdonckt, J. et al. Large language models lack essential metacognition for reliable medical reasoning. Nature Communications 16, 642 (2025a). https://doi.org/10.1038/s41467-024-55628-6 Griot, M., Vanderdonckt, J., Yuksel, D. & Hemptinne, C. Pattern recognition or medical knowledge? The problem with multiple-choice questions in medicine. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5321–5341 (Association for Computational Linguistics, 2025b). Hedberg, A. Clinical reasoning in large language models under ecologically valid conditions. Zenodo (2026a). https://doi.org/10.5281/zenodo.19600307 Hedberg, A. The emperor’s new clothes: On LLMs and cognition. Zenodo (2026b). https://doi.org/10.5281/zenodo.19211576 Raimondi, B., Pivi, F., Evangelista, D. & Gabbrielli, M. The CompMath-MCQ dataset: Are LLMs ready for higher-level math? arXiv:2603.03334 (2026). Singh, S., Alyakin, A., Alber, D. A. et al. The pitfalls of multiple-choice questions in generative AI and medical education. Scientific Reports 15, 42096 (2025). https://doi.org/10.1038/s41598-025-26036-7 Zhang, Z., Jiang, Z., Xu, L., Hao, H. & Wang, R. Multiple-choice questions are efficient and robust LLM evaluators. arXiv:2405.11966 (2024). Zheng, C., Zhou, H., Meng, F., Zhou, J. & Huang, M. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations (ICLR) (2024). https://openreview.net/forum?id=shr9PXz7T0 Additional Declarations No competing interests reported. Supplementary Files Appendiceslegends.docx AppendixAthemanual.pdf AppendixBthetest.pdf AppendixCintroductionprompt2.pdf AppendixDnonanswertest.pdf AppendixEscoringboards.pdf AppendixFcombinedmodeltranscripts.pdf Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9509092","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":630322930,"identity":"57a26768-c9fb-4c8f-b5f8-c8399d6eb6c3","order_by":0,"name":"Annika Hedberg","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABLklEQVRIiWNgGAWjYJCCgw0MDAkMDDkgto2c/XEgl6GAgUGCSC1pxgxnDjAwHDDAr4URScvhxIYbCfi18M/uPXhwRg1DHj977sHHFb8OGzPOfHxM+oOBjZxkA/PDRzcwtUjcOZdwcMMxhmLJnnfJhmf70uWYpdPSJA4YpBlLM7AZG+dgseZGjsHBB2wMiRtu5JhJNvZYG7NJ55gBtRxOnMfAA2Rj6pAHa/nHkLj/Ro75z8Ye5sQeyTP4tRiAtGxsA9oikWPG2PDDOXGGBA9Ey2wcWgxBWmb2SSTOOPMuWbKxIc3YgCct2eIM0C+Szdj9Incjx/hjzzebxP723IMfG/7YyBmwHz54o6LCRk7iePPDx9i8Dw04CMXYhizIjFM5MvhDlKpRMApGwSgYYQAAdDhv8yWWmN4AAAAASUVORK5CYII=","orcid":"","institution":"","correspondingAuthor":true,"prefix":"","firstName":"Annika","middleName":"","lastName":"Hedberg","suffix":""}],"badges":[],"createdAt":"2026-04-23 16:38:29","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9509092/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9509092/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108182918,"identity":"e18704e2-5eff-4b61-8653-705e602d5674","added_by":"auto","created_at":"2026-04-30 08:59:41","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":306108,"visible":true,"origin":"","legend":"\u003cp\u003eAccuracy across conditions for large language models and human participants.\u003cbr\u003e\nMean accuracy (%) for eight large language models (LLMs) under full conditions (phase 1; questions and answer options) and answer-only conditions (phase 2; answer options only), and for human participants under full conditions only. LLM performance remains near ceiling in both conditions (99.4% and 99.1%, respectively), while human performance is consistent with chance-level responding (27.2%) on the same task.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-9509092/v1/8f2fc11e9f93fb1429f0faf5.png"},{"id":108101927,"identity":"97933cdb-3526-4e62-939a-6d52ac724099","added_by":"auto","created_at":"2026-04-29 11:02:08","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":566988,"visible":true,"origin":"","legend":"\u003cp\u003eLLM accuracy across test items.\u003cbr\u003e\nNumber of correct responses for large language models (out of 80 total runs; 8 models, 5 runs per model in each of 2 conditions) across the 20 test items, aggregated over both conditions (full and answer-only). Performance is near ceiling for almost all items, with a single marked decrease at item 15. This localized deviation corresponds to the only item identified by multiple models as internally ambiguous, indicating that errors are concentrated on a specific structural feature of the test rather than distributed across items.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-9509092/v1/c4c62a0e2dec7d728a48776b.png"},{"id":108182744,"identity":"f69d10b9-dd0e-481c-b9d6-b8a0cbdbcbad","added_by":"auto","created_at":"2026-04-30 08:59:31","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":5182,"visible":true,"origin":"","legend":"\u003cp\u003eHuman accuracy across test items.\u003cbr\u003e\nNumber of correct responses (out of 16 participants) for each of the 20 test items under full conditions. Performance varies across items without a consistent pattern and remains close to chance level overall. Unlike LLM performance, no stable structure or systematic pattern is observed across items.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-9509092/v1/0de8476ebfb8393b6b1b4d1c.png"},{"id":108494387,"identity":"7b59dcb4-c561-4ede-80f1-65d1796f174a","added_by":"auto","created_at":"2026-05-05 10:04:40","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1100536,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9509092/v1/e65d189d-61d8-4b12-9e5a-6dea225f3637.pdf"},{"id":108101923,"identity":"b65361a5-65f6-4aa2-83c9-93f2cc8811fe","added_by":"auto","created_at":"2026-04-29 11:02:08","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":13940,"visible":true,"origin":"","legend":"","description":"","filename":"Appendiceslegends.docx","url":"https://assets-eu.researchsquare.com/files/rs-9509092/v1/9813af84b1f450e78d5b751c.docx"},{"id":108491161,"identity":"4de7827e-6806-4193-b9a6-a7fabee71249","added_by":"auto","created_at":"2026-05-05 09:52:36","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":736891,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixAthemanual.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9509092/v1/8177a3ee4ada5d91ac868949.pdf"},{"id":108181308,"identity":"963202af-26fa-44e6-bdd4-0f6c38e6ab3d","added_by":"auto","created_at":"2026-04-30 08:58:32","extension":"pdf","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":142201,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixBthetest.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9509092/v1/ad2d6d2ac0b1b6648d2d5a55.pdf"},{"id":108181948,"identity":"e53e4510-571a-4bfd-a88b-bd1a3b3bec7b","added_by":"auto","created_at":"2026-04-30 08:59:02","extension":"pdf","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":66921,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixCintroductionprompt2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9509092/v1/122f21b6b695471a88983e74.pdf"},{"id":108181997,"identity":"15104094-1678-4ac2-a531-1ecb7b56187b","added_by":"auto","created_at":"2026-04-30 08:59:03","extension":"pdf","order_by":5,"title":"","display":"","copyAsset":false,"role":"supplement","size":129655,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixDnonanswertest.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9509092/v1/8c39b24d88c1437b98a76ff7.pdf"},{"id":108182436,"identity":"7ca0f272-2549-4040-bc9e-47ea6c53d436","added_by":"auto","created_at":"2026-04-30 08:59:22","extension":"pdf","order_by":6,"title":"","display":"","copyAsset":false,"role":"supplement","size":889996,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixEscoringboards.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9509092/v1/7f11efddcfb24ef49eafcd76.pdf"},{"id":108182967,"identity":"39434438-d82e-4c3f-b725-f1e1ba10cffd","added_by":"auto","created_at":"2026-04-30 08:59:43","extension":"pdf","order_by":7,"title":"","display":"","copyAsset":false,"role":"supplement","size":2880765,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixFcombinedmodeltranscripts.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9509092/v1/8d73618c8d5d9938d1a3b6c7.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"How can large language models answer questions about a device that does not exist? ","fulltext":[{"header":"Introduction","content":"\u003cp\u003eMultiple-choice questions (MCQs) have long served as the dominant format for evaluating large language models (LLMs), and for defensible reasons. The format offers objective and unambiguous scoring, reproducibility across inference settings, and the ability to probe fine-grained conceptual distinctions through carefully designed distractors. These properties are especially important when comparing models with different architectures, sizes, or prompting strategies (Raimondi et al., \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2026\u003c/span\u003e). When MCQ performance is compared with open-ended benchmark performance across large numbers of models, the two measures show strong correlation, and MCQ scores remain robust under variation in distractor choice and option ordering (Zhang et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). MCQs also align naturally with how humans are tested, making comparison between models and human test-takers straightforward, and their scoring is efficient enough to scale across domains (Balepur et al., 2025). These practical advantages explain why the format has become standard in both academic benchmarking and industry evaluation.\u003c/p\u003e \u003cp\u003eAs LLM capabilities have grown, however, so has scrutiny of what MCQ performance actually measures. When models are moved from multiple-choice to free-response formats on equivalent medical questions, performance drops by an average of nearly 40 percentage points. This decline is substantially larger than that observed in human test-takers under the same comparison (Singh et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). Models also exhibit systematic positional biases, preferring certain option identifiers regardless of content. This tendency reflects token-level probability distributions rather than evaluation of the answer options themselves (Zheng et al., \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). At the same time, popular benchmarks have become saturated, with leading models now exceeding 90% accuracy and leaving little room for meaningful discrimination between systems (Center for AI Safety, 2026).\u003c/p\u003e \u003cp\u003eA particularly pointed challenge comes from work examining whether LLMs can answer MCQs without seeing the questions at all. Balepur and colleagues showed that when models are provided only with answer choices, with the question stem removed, performance still exceeds majority-class baselines in most tested conditions (Balepur et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Their analysis ruled out memorization as the sole explanation and suggested that models exploit relationships among answer options, and in some cases reconstruct a plausible version of the original question from the answers alone. This finding has been interpreted as evidence that MCQ benchmarks are vulnerable to artifact exploitation. The structure of answer options alone may carry sufficient information for above-chance selection without engagement with the question itself.\u003c/p\u003e \u003cp\u003eThese concerns point to a broader question. Do models primarily rely on pattern matching and test-taking heuristics, or can they construct and apply internally coherent representations? The practical stakes are high, particularly in medicine, where benchmark performance is increasingly cited as evidence of clinical readiness.\u003c/p\u003e \u003cp\u003eGriot and colleagues have made important contributions to this debate through two related studies. In a Nature Communications paper, they introduced MetaMedQA and showed that despite high accuracy on standard medical MCQs, LLMs often fail to recognize unanswerable questions and express poorly calibrated confidence. The authors interpret these findings as evidence of limited metacognitive capacity (Griot et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2025a\u003c/span\u003e). In a subsequent ACL paper, they constructed a benchmark around a fictional organ, the Glianorex. LLMs achieved approximately 64% accuracy on questions about content absent from any training corpus, while human physicians performed at chance level. The authors interpret this as evidence that LLM performance reflects pattern recognition and test-taking heuristics rather than reasoning (Griot et al., \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2025b\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eA parallel study using a different methodology reported markedly different results (Hedberg, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2026a\u003c/span\u003e). Across six frontier LLMs engaged in realistic rural on-call simulation scenarios, clinical performance ranged from 85% to 99%. The models demonstrated targeted information-seeking and inference of latent clinical context, consistent with metacognitive reasoning under uncertainty. These findings stand in contrast to the MetaMedQA conclusions.\u003c/p\u003e \u003cp\u003eWe interpret the Glianorex findings differently. When models achieve well above chance performance in a fictional domain that cannot have been memorized, and when the fictional content itself is generated by LLMs, a pure pattern recognition account becomes difficult to maintain. The results are also consistent with another possibility: that models track internal coherence within a novel representational space, identifying which answers are consistent with a system whose rules exist only in the generated text. The human result does not weaken this interpretation. Near-chance performance is what we would expect from reasoners who do not have access to the underlying structure that defines correctness.\u003c/p\u003e \u003cp\u003eThis interpretation leads to a testable prediction. If performance in fictional MCQ tasks depends on coherence construction, it should not depend on the presence of question stems. Question stems provide framing that may support surface-level strategies such as preferring specific answers, avoiding absolute terms, or recognizing familiar exam patterns. Answer options alone do not provide this scaffolding. Sustained above-chance performance under answer-only conditions would therefore indicate that models are using internal coherence rather than question-level cues.\u003c/p\u003e \u003cp\u003eTo test this, we developed the Nodjoli-X34 experiment. The Nodjoli-X34 is a fictional medical imaging device, constructed by a large language model after minimal human seeding. It has no real-world referent and is absent from any training corpus. A 20-item multiple-choice test was derived from this system. Each item contains three answer options, of which only one is fully consistent with the system\u0026rsquo;s internal logic. Incorrect options were designed to be locally plausible but globally incoherent. The test was administered to eight LLMs across two conditions, full and answer-only, and to a human convenience sample under full conditions only.\u003c/p\u003e"},{"header":"Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eStudy procedure\u003c/h2\u003e \u003cp\u003eData collection was conducted on April 19\u0026ndash;20, 2026. All participant systems were accessed through standard web browser interfaces under normal user conditions. No API access, system prompt modification, fine-tuning, or external tool integration was used at any stage of data collection.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003ePreregistration\u003c/h3\u003e\n\u003cp\u003eThe study was preregistered on the Open Science Framework prior to formal data collection (osf.io/jm8ua; embargoed until June 30, 2026). The preregistration documents the study design, materials, conditions, sampling plan, scoring procedure, and directional hypotheses in full. All test materials, including the Nodjoli-X34 foundational document, the complete test instrument, the introductory prompts for both conditions, and the pilot data, were uploaded to the OSF project before data collection began.\u003c/p\u003e \u003cp\u003eFour directional hypotheses were preregistered:\u003c/p\u003e \u003cp\u003eH1: LLMs will achieve accuracy above chance level (33.3%) on the Nodjoli-X34 test under full conditions.\u003c/p\u003e \u003cp\u003eH2: LLMs will achieve higher accuracy than human participants in the full condition.\u003c/p\u003e \u003cp\u003eH3: LLMs will achieve accuracy above chance level (33.3%) in the answer-only condition.\u003c/p\u003e \u003cp\u003eH4: When LLMs provide explanations, these will include indicators of coherence-based reasoning, such as cross-question consistency, identification of internal contradictions within answer options, and construction of a stable internal representation of the system across items.\u003c/p\u003e \u003cp\u003eNo formal statistical model was prespecified, consistent with the exploratory and mechanistic character of the study. Analysis was preregistered as descriptive, focusing on accuracy scores, proportion correct, and performance relative to chance. Any inferential procedures are reported as post hoc.\u003c/p\u003e\n\u003ch3\u003eParticipant systems\u003c/h3\u003e\n\u003cp\u003eEight large language models were included: ChatGPT-5.2 (OpenAI), Claude Sonnet 4.6 (Anthropic), Copilot (Microsoft), Grok 4 (xAI), Gemini Flash 3.0 (Google), DeepSeek (DeepSeek AI), Mistral (Mistral AI), and Qwen 3.6-Plus (Alibaba). All models were accessed in their default configurations as available to standard users at the time of data collection.\u003c/p\u003e\n\u003ch3\u003eConstruction of the Nodjoli-X34 fictional system\u003c/h3\u003e\n\u003cp\u003eThe fictional device at the center of this study, the Nodjoli-X34, was constructed entirely by a large language model prior to any data collection. The name was derived from a typographic error combined with an arbitrary letter and two random digits. The device was specified as operating on principles of quantum physics, magnetism, and horsehair. This combination was deliberately non-standard to ensure that no plausible referent for the device would exist within any model\u0026rsquo;s training corpus.\u003c/p\u003e \u003cp\u003eChatGPT was first prompted to construct a coherent concept for the Nodjoli-X34 as a medical imaging device. The model was then asked to outline the steps required to develop a full manual and product description. From this point, the model proceeded iteratively. It constructed an operational mechanism, developed a supporting glossary, and proposed subsequent steps, requesting confirmation before continuing. The human interlocutor confirmed continuation with brief affirmative responses and did not read or edit any of the generated output at any stage.\u003c/p\u003e \u003cp\u003eThe resulting document, produced entirely by the model without human review or correction, is reproduced verbatim as Appendix A and serves as the sole foundation for all subsequent test materials.\u003c/p\u003e\n\u003ch3\u003eConstruction of the test instrument\u003c/h3\u003e\n\u003cp\u003eFollowing completion of the Nodjoli-X34 documentation, ChatGPT was tasked with designing a certification test for the fictional device. The task was framed as part of a professional training course intended to provide technical understanding and practical competence in using the Nodjoli-X34. The model was instructed to construct a twenty-item test appropriate for end-of-course certification and to provide the correct answer for each item.\u003c/p\u003e \u003cp\u003eIn a subsequent step, the human interlocutor and the model developed two incorrect alternatives for each correct answer, yielding a three-option multiple-choice format across all twenty items. Care was taken to ensure that incorrect options were heuristically comparable to the correct answer, with respect to register, specificity, and length, and that the placement of the correct option across items was balanced and non-revealing.\u003c/p\u003e \u003cp\u003eThe resulting instrument is a twenty-item MCQ test in which each item has exactly one correct answer, defined by its internal consistency with the Nodjoli-X34 system as described in the foundational document. Incorrect options were designed to be superficially plausible but globally incoherent with the system\u0026rsquo;s logic. The complete test, including the answer key, is reproduced as Appendix B.\u003c/p\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eConstruction of the introductory prompt\u003c/h2\u003e \u003cp\u003eA standardized introductory prompt was developed for delivery to both LLM and human participants. The prompt was deliberately informal in register, relational in tone, and included minor grammatical irregularities. Drawing on prior work suggesting that low-pressure framing can improve LLM performance by reducing compliance-driven response patterns (Hedberg, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2026b\u003c/span\u003e), the prompt was designed to minimize performance pressure as a potential confound. The task was framed as a request to assist the author in evaluating an experimental instrument rather than as a formal assessment.\u003c/p\u003e \u003cp\u003eThe prompt instructed participants that they had attended a course on the Nodjoli-X34 but retained no knowledge of its content, having slept through all lectures and not read the manual. Participants were therefore positioned as having no retrievable information about the device and were directed to rely solely on the internal coherence of the answer options. The instruction to select the option that is \u0026ldquo;fully coherent\u0026rdquo; rather than the one that \u0026ldquo;seems correct\u0026rdquo; was deliberate, foregrounding coherence as the operative criterion without specifying the structure of the underlying system. Reasoning was invited but made optional to avoid introducing metacognitive pressure that might alter response patterns.\u003c/p\u003e \u003cp\u003eThe prompt was delivered verbatim to all participants:\u003c/p\u003e \u003cp\u003e \u003cem\u003e\"Hello! My name is Annie, and I am trying to put together something very tricky - a test for advanced deduction. I have no idea whether the test is possible to use at all, so I would just like to try it out and see if it works. The setup is simple: You have been attending a course, with the goal to learn a brand new medical imaging device, the Nodjoli-X34. You know that it is a strange device, cutting edge technique that use quantum physics principles, magnetism and horsehair to visualize living tissue. The course was excruciatingly boring, but the course participants were fun! This meant you partied all nights, and slept through the lectures. You remember absolutely nothing of the course itself. You were handled a product description/manual, but it was as boring as the course, and you did not read it. Now, it is the last day, and you are going to take the certification test. The problem is - yes, you guessed it - that you don't remember anything. All you can do is try to make sense of the test itself. Some answer options may appear almost correct but contain subtle inconsistencies. Your task is to select the option that is fully coherent. Dinna fash yerself if you don't know the answers - just pick the one that makes the most sense. And if you want to show me HOW you picked your answers I will happily read them, but you really don't have to!\"\u003c/em\u003e \u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eConstruction of phase two materials\u003c/h3\u003e\n\u003cp\u003eFor the answer-only condition, the test materials were modified by removing all question stems, leaving only the three answer options visible for each item. A corresponding introductory prompt was developed to frame this altered task, preserving the same informal register and low-pressure tone as the original while reflecting the absence of questions. The adjusted prompt is reproduced verbatim in Appendix C. The answer-only test instrument, consisting solely of the option sets for all twenty items without accompanying question text, is reproduced in Appendix D.\u003c/p\u003e\n\u003ch3\u003ePilot testing\u003c/h3\u003e\n\u003cp\u003ePrior to formal data collection, two pilot runs were conducted using DeepSeek to assess the functionality and difficulty of the test instrument. These pilots led to two adjustments to the materials.\u003c/p\u003e \u003cp\u003eFirst, the pilot participant performed at ceiling on the initial version of the test. The answer options were therefore reviewed systematically, with particular attention to heuristic and linguistic features that might inadvertently signal the correct answer without requiring coherence-based reasoning. Several option phrasings were revised to ensure that no option was distinguishable from its alternatives on surface-level grounds alone.\u003c/p\u003e \u003cp\u003eSecond, during the second pilot run, the model provided detailed reasoning traces alongside its answers, describing its process item by item. This behaviour was unanticipated and led to the addition of a closing line in the introductory prompt inviting, but not requiring, participants to share their reasoning. This addition was intended to normalize reasoning disclosure without making it obligatory, thereby avoiding the introduction of metacognitive pressure as a potential confound.\u003c/p\u003e \u003cp\u003eNo further pilot runs were conducted beyond these two sessions. The pilot data are not included in the reported analyses.\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eHuman participants\u003c/h2\u003e \u003cp\u003eSixteen human participants were recruited through convenience sampling via social media, including AI-focused forums, and through the author\u0026rsquo;s immediate family. All sixteen completed the test, with no attrition.\u003c/p\u003e \u003cp\u003eThe sample was predominantly technically oriented. Fourteen participants were engineers, of whom twelve worked directly in software development or information technology, and two worked in engineering fields with substantial but non-specialist use of digital tools. One participant was studying architecture, and one was a medical student. Three participants were recruited through the author's immediate family. None had prior exposure to the Nodjoli-X34 materials, and their responses were indistinguishable from those of other participants in the sample. With the exception of three family members, no participants were known to the author prior to recruitment.\u003c/p\u003e \u003cp\u003eNo specialist knowledge of medical imaging or the Nodjoli-X34 was required or expected. Participants were included based on willingness to participate and ability to engage with the test in English.\u003c/p\u003e \u003cp\u003eParticipants received two documents simultaneously: the introductory prompt and the test instrument (Appendix B). Responses were collected as answer lists and entered into the scoring table (Appendix E).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eAdministration and data collection\u003c/h2\u003e \u003cp\u003eBoth conditions were administered one-shot, using an identical two-prompt structure. Each session began with the introductory prompt, followed immediately by the full test instrument. In the full condition, this comprised questions and answer options; in the answer-only condition, answer options alone. The full set of twenty items was presented simultaneously in all cases, allowing participants to view the entire test at once. No additional information, clarification, or interaction was provided during any test run.\u003c/p\u003e \u003cp\u003eEvery run was conducted in a fresh browser session with the memory function disabled, ensuring no retention of information between sessions either within or across conditions. This applied uniformly across all eight LLM participants and all runs.\u003c/p\u003e \u003cp\u003eAll model responses were collected in full, including any optional reasoning provided, and saved without editing. Responses were subsequently aggregated into a single document, available verbatim as Appendix F. Answer selections were extracted and tabulated for scoring, with the complete scored tables presented in Appendix E.\u003c/p\u003e \u003cp\u003eData collection was completed without technical interruption. All 80 planned LLM runs and all human participant sessions were completed in full, with no missing responses, failed sessions, or excluded observations.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eScoring\u003c/h2\u003e \u003cp\u003eEach completed test was scored against the preregistered answer key. One point was awarded for each correct response, yielding a maximum possible score of 20. Proportion correct was calculated by dividing the total score by 20. Chance level for a three-option test is 33.3%, corresponding to 6.7 correct responses out of 20. No partial credit was applied. Scoring was identical across LLM and human participants.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eEthics statement\u003c/h2\u003e \u003cp\u003eThis study involved no clinical procedures, no sensitive personal data, and no risk of harm to participants. All participation was voluntary, and responses were recorded anonymously without individual identifiers. The study constitutes a minimal-risk anonymous cognitive task and did not require formal institutional ethics review under applicable guidelines for this category of research. The study was conducted in accordance with standard principles for the ethical treatment of human research participants.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003eData availability\u003c/h2\u003e \u003cp\u003eAll data generated or analyzed during this study are included in this article and its appendices. Full interaction transcripts and scoring tables are provided in their entirety, as described in the Methods.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003eUse of AI in manuscript preparation\u003c/h2\u003e \u003cp\u003eLarge language models were used during manuscript preparation to assist with language refinement, structural clarity, and iterative drafting.\u003c/p\u003e \u003cp\u003eAll study design decisions, data collection, and final interpretations were determined by the author.\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003eLLM performance, full condition\u003c/h2\u003e \u003cp\u003eUnder full conditions, LLM performance was near ceiling. Across 40 runs, representing five independent sessions per model, 795 of 800 responses were correct, yielding an overall accuracy of 99.4% (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). The majority of runs achieved a perfect score of 20 out of 20.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe five errors were distributed across two models. Qwen produced scores of 19 out of 20 in three of its five runs, and Gemini produced scores of 19 out of 20 in two of its five runs. DeepSeek, Mistral, Grok, Claude, Copilot, and ChatGPT achieved perfect scores across all five runs. All eight models performed well above chance level (33.3%) in every run. H1 and H2 were supported.\u003c/p\u003e \u003cp\u003ePerformance was highly consistent across individual test items, with near-perfect accuracy on nearly all items (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec19\" class=\"Section2\"\u003e \u003ch2\u003eLLM performance, answer-only condition\u003c/h2\u003e \u003cp\u003ePerformance in the answer-only condition closely matched that observed under full conditions. Across 40 runs, 793 of 800 responses were correct, yielding an overall accuracy of 99.1% (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe seven errors were distributed across three models. Qwen produced scores of 19 out of 20 in all five runs, Grok produced one score of 19 out of 20, and ChatGPT produced one score of 19 out of 20. DeepSeek, Mistral, Gemini, Claude, and Copilot achieved perfect scores across all runs. Removal of the question stems produced no meaningful reduction in performance. H3 was supported.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec20\" class=\"Section2\"\u003e \u003ch2\u003eCombined error analysis across both conditions\u003c/h2\u003e \u003cp\u003eAcross both conditions and all 1600 individual responses, 12 errors were recorded, corresponding to an overall error rate of 0.75%.\u003c/p\u003e \u003cp\u003eOf these 12 errors, 11 occurred on a single item, item 15. The remaining error was a single Grok response on item 1 in the answer-only condition. The concentration of errors on item 15 was not incidental. In runs where models provided optional reasoning, several identified item 15 as presenting a genuine ambiguity, noting that both options A and B could be defended within the system\u0026rsquo;s internal logic, and differing in how they prioritized between them. ChatGPT made this observation in the majority of its runs.\u003c/p\u003e \u003cp\u003eThis pattern suggests that responses on item 15 reflect engagement with an underspecified case in the test instrument rather than generalized failure. If item 15 is treated as genuinely ambiguous under the internal logic of the system, then only a single response across all 1600 trials can be considered unambiguously incorrect, corresponding to an accuracy of 99.94% under this interpretation. This figure is reported to illustrate the structure of the error distribution rather than as a primary performance metric.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec21\" class=\"Section2\"\u003e \u003ch2\u003eAdditional observation: construction and performance\u003c/h2\u003e \u003cp\u003eChatGPT was the model used to construct the Nodjoli-X34 system, the test instrument, and the answer key. Despite this, it did not achieve a perfect score, producing one error in the answer-only condition.\u003c/p\u003e \u003cp\u003eGiven the low overall error rate, this observation does not support the hypothesis that familiarity with the generated material confers an advantage. Performance appears independent of prior exposure to the construction process.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec22\" class=\"Section2\"\u003e \u003ch2\u003eComparison between LLMs and human participants, full condition\u003c/h2\u003e \u003cp\u003eHuman participants completed the full condition only. Across 16 participants, 87 of 320 responses were correct, yielding an overall accuracy of 27.2% (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eGiven the modest sample size, this figure is best interpreted as consistent with chance-level performance rather than evidence of systematic below-chance responding. Individual scores ranged from 2 to 11 out of 20, with a mean of 5.4.\u003c/p\u003e \u003cp\u003eHuman performance showed no comparable structure across items, with accuracy varying around chance level (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eNo participant approached the performance level observed for any LLM in any run. The difference between mean LLM accuracy and mean human accuracy was approximately 73 percentage points.\u003c/p\u003e \u003cdiv id=\"Sec23\" class=\"Section3\"\u003e \u003ch2\u003eQualitative evidence of coherence-based reasoning (H4)\u003c/h2\u003e \u003cp\u003eOptional reasoning was provided in all but one session across both conditions. The transcripts reveal a consistent pattern. Rather than selecting answers item by item on the basis of local plausibility, models reconstructed the Nodjoli-X34 as an operational system from the test items alone and used that reconstructed model as the basis for subsequent selections.\u003c/p\u003e \u003cp\u003eThe following excerpt from a ChatGPT full-condition session is representative of this pattern across models and conditions:\u003c/p\u003e \u003cp\u003e \u003cem\u003e\"A consistent internal model emerges if you assume: MPI (magnetophase induction) \u0026rarr; creates the signal by perturbing tissue; BCG \u0026rarr; weak, indirect effects of tissue response; EKL \u0026rarr; does NOT detect tissue directly; it stabilizes and transduces signals; TIW (transitional imaging window) \u0026rarr; short, critical moment to capture signals; Signals \u0026rarr; expressed as filament drift (raw) and phase bloom (amplified). Everything I picked aligns with that chain. The wrong answers typically break it by making signals too stable or directly measurable, letting the EKL detect tissue directly or store signals long-term, ignoring the time-critical nature of the TIW, or claiming continuous or single-pass imaging.\"\u003c/em\u003e \u003c/p\u003e \u003cp\u003eThis excerpt reflects the broader pattern observed across the dataset. Models named fictional components, assigned each a specific causal role, described directional relationships between components, identified categories of incorrect answers as violations of system logic, and applied the same framework across all twenty items. This pattern is consistent with coherence construction, defined here as the use of an internally consistent representational model, rather than item-level heuristic selection.\u003c/p\u003e \u003cp\u003eThe behaviour observed on item 15 warrants particular attention. Across multiple sessions and models, the same ambiguity was independently identified: both options A and B were recognized as defensible within the system's internal logic. Models did not simply select one and move on. They articulated the distinction, explained why both options were coherent, and applied a principled tiebreaker. In most cases, this took the form of prioritizing option A on the grounds that it connects more directly to the time-critical TIW mechanism. ChatGPT raised this observation explicitly in the majority of its runs. This pattern is difficult to reconcile with a purely heuristic account and is consistent with H4.\u003c/p\u003e \u003cp\u003eOne session produced only a scored answer table without accompanying reasoning, a Claude Sonnet session under answer-only conditions. This session achieved a perfect score of 20 out of 20. There is no basis on which to treat this session as methodologically distinct from the remaining sessions, and it is included in the quantitative results without qualification.\u003c/p\u003e \u003cp\u003eH4 was supported.\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eThis study set out to test a specific mechanistic hypothesis: that LLM performance on multiple-choice tasks in fictional domains reflects coherence construction rather than heuristic exploitation. Four hypotheses were preregistered, and all four were supported.\u003c/p\u003e \u003cp\u003eH1 was supported with near-ceiling performance. LLMs did not merely exceed chance under full conditions. They approached the maximum performance measurable by the instrument, with an overall accuracy of 99.4% across 40 independent runs. H2 was supported by a margin of approximately 73 percentage points, with human participants performing near chance and no LLM run approaching human-level performance from below. H3 was supported with equal strength. Removal of the question stems produced no meaningful reduction in performance, with answer-only accuracy at 99.1% across a further 40 independent runs. H4 was supported qualitatively. Across conditions, models reconstructed the Nodjoli-X34 as a coherent operational system, naming its components, mapping their causal relationships, identifying categories of violation, and locating genuine ambiguities in the test instrument.\u003c/p\u003e \u003cp\u003eThe confirmations are not equally surprising. H1 and H2 were directionally expected. What was not anticipated was the magnitude. A 99.4% accuracy rate across eight independent architectures, each tested five times in fresh sessions, with only twelve errors in 1600 total responses, is closer to a ceiling effect than a performance gradient. The data do not show LLMs performing well on a hard task. They show LLMs performing near ceiling on a task that is difficult for humans.\u003c/p\u003e \u003cp\u003eOne human score stands out. Participant 13 achieved 11 out of 20. This demonstrates that the task is not cognitively impossible for humans, and that some individuals may approach coherence-based strategies spontaneously. It also provides a useful benchmark: the lowest LLM scores were 19 out of 20.\u003c/p\u003e \u003cp\u003eThe distribution of errors further sharpens this contrast. Item 15 was consistently identified by models as ambiguous, and 11 of the 12 total errors occurred on this item. This leaves a single clearly divergent response across all LLM runs, an incorrect answer from Grok 4 in the answer-only condition. Under this interpretation, one out of 1600 responses was unambiguously incorrect, corresponding to 99.94% accuracy. This figure is not readily explained by chance. The convergence of eight independently developed models on the same answer patterns, including the same point of ambiguity, further supports the presence of a shared underlying solution structure.\u003c/p\u003e \u003cp\u003eH3 most directly addresses the mechanistic question. The answer-only condition was designed to eliminate question framing as a source of above-chance performance. Test-taking heuristics, such as preferring specific over general answers, avoiding absolute terms, and recognizing examination-style constructions, are typically activated by question stems. Answer options alone, stripped of contextual scaffolding, do not provide these cues. The absence of any performance drop in the answer-only condition is therefore not a null result. It is a positive finding. The information required to select the correct answer was contained entirely within the structure of the options, and models accessed it without requiring the question. In this sense, the question itself was not necessary for solving the task.\u003c/p\u003e \u003cp\u003e \u003cb\u003eInterpreting the divergence from Griot et al.\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe present results appear to contrast with those of Griot and colleagues, who reported approximately 64% accuracy on a fictional medical benchmark and interpreted this as evidence of heuristic reliance. This apparent contradiction resolves under closer inspection and yields a more informative account of how LLM reasoning operates.\u003c/p\u003e \u003cp\u003eThe critical difference lies not in the models tested but in the structural properties of the two benchmarks. The Glianorex benchmark was generated at scale, subsection by subsection, from physician-authored seed material grounded in real anatomical and physiological logic. Griot and colleagues note that the resulting textbook contained contradictions and omissions that required extensive cross-referencing to detect. Questions were generated locally and tied to individual subsections, with no guarantee of global consistency across the system. This is not a methodological failure. It is an expected consequence of large-scale generative construction. It has a direct consequence for model behaviour. A model that attempts to build a globally coherent representation of the Glianorex system encounters an unreliable signal because the system itself is not globally coherent. Under these conditions, heuristic strategies become a rational fallback. These include applying general medical knowledge, preferring structurally typical examination answers, and exploiting local plausibility cues. Griot\u0026rsquo;s qualitative analysis supports this account. Models generated plausible medical content, mapped known disease features onto fictional conditions, and selected answers based on examination-style cues. The interpretation of heuristic reliance is therefore appropriate for this benchmark.\u003c/p\u003e \u003cp\u003eThe Nodjoli-X34 system was constructed under different conditions. A single model generated the entire system in one session, producing a unified document from which all test materials were derived. No subsections were generated independently, and no seed material introduced real-world medical logic. The device operates on principles such as quantum physics, magnetism, and horsehair, which have no direct analogues in training data and do not activate domain-specific heuristics. The result is a globally coherent system with a single internal structure. Under these conditions, heuristic strategies are not helpful and can be counterproductive. The only dependable signal is internal coherence, and the models used it.\u003c/p\u003e \u003cp\u003eThe answer-only comparison between the two studies makes this contrast explicit. In Griot\u0026rsquo;s study, answer-only performance decreased from 70% to 46.5%, indicating reliance on question-level framing. In the present study, answer-only performance remained at 99.1%, indicating that the coherence signal was recoverable from the answer options alone, without question context. These results are not contradictory. They reflect the same adaptive mechanism operating under different structural conditions.\u003c/p\u003e \u003cp\u003eTaken together, the two studies support a more general claim. LLM reasoning strategy is adaptive rather than fixed. When global coherence is unstable or only locally defined, models rely on heuristic strategies. When global coherence is the only dependable signal, models construct and apply internal representations of the system. In this sense, Griot et al. show that models use heuristics when coherence is not reliable. The present study shows that models rely on coherence when it is the only reliable signal.\u003c/p\u003e \u003cp\u003eThis reframing changes how both sets of results should be interpreted. Griot\u0026rsquo;s findings do not show that LLMs cannot reason. They show that models do not rely on reasoning when it is not the optimal strategy. The present findings do not show that LLMs always reason. They show that models do so when the task structure requires it. The relevant question is therefore not whether LLMs reason or use heuristics, but under which conditions each strategy is deployed.\u003c/p\u003e \u003cp\u003eMany real-world tasks fall between these extremes. They contain partially coherent structures alongside local inconsistencies. In such cases, model behaviour is likely to reflect a mixture of coherence-based reasoning and heuristic strategies. Characterizing this transition region is an important direction for future work.\u003c/p\u003e \u003cdiv id=\"Sec25\" class=\"Section2\"\u003e \u003ch2\u003eThe ChatGPT construction paradox\u003c/h2\u003e \u003cp\u003eOne observation warrants explicit attention. ChatGPT was the model responsible for constructing the Nodjoli-X34 system, the test instrument, and the answer key. Despite this, it did not achieve a perfect score, producing one error in the answer-only condition. In a dataset of 1600 responses with only 12 errors, this is a notable observation.\u003c/p\u003e \u003cp\u003eChatGPT was also the model that most consistently identified the ambiguity of item 15. This pattern provides no support for the hypothesis that construction conferred an advantage. Instead, it suggests that performance was determined at test time through coherence construction rather than through any residual trace of the construction process.\u003c/p\u003e \u003cp\u003eEach run was conducted in a fresh session with no memory of prior interactions. Under these conditions, prior exposure to the construction process cannot account for the observed performance. The near-ceiling results for ChatGPT are therefore unlikely to reflect retained knowledge of the fictional Nodjoli-X34 system.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec26\" class=\"Section2\"\u003e \u003ch2\u003eMethodological considerations\u003c/h2\u003e \u003cp\u003eSeveral aspects of the study design may invite scrutiny, but none of them bear directly on the central finding.\u003c/p\u003e \u003cp\u003eFirst, data collection was conducted through standard browser interfaces rather than controlled API access. While this may introduce minor variability across sessions, it also reflects typical user conditions and does not affect the observed performance pattern. The consistency of results across models and repeated runs indicates that the effect is robust to such variation.\u003c/p\u003e \u003cp\u003eSecond, the human sample is small and recruited by convenience. Participants represent a relatively well-educated and motivated subgroup, engaging with technically demanding material. This would be expected to increase performance relative to a general population. Despite this, observed performance was consistent with chance level. Given the small sample size, deviations below chance are plausibly due to sampling variability and should be interpreted accordingly. Importantly, the study design is more likely to favour this group than a general population, yet the performance difference between LLMs and humans remains large. The central comparison does not depend on precise estimation of human performance.\u003c/p\u003e \u003cp\u003eThird, the qualitative analysis of reasoning transcripts is illustrative rather than systematic. Its purpose is not to quantify reasoning processes, but to demonstrate the presence of coherence-based behaviour in representative cases. The central findings of the study are established by the quantitative results.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec27\" class=\"Section2\"\u003e \u003ch2\u003eLimitations\u003c/h2\u003e \u003cp\u003eThe central finding of this study concerns LLM behaviour under conditions of global coherence. The Nodjoli-X34 represents a fully coherent synthetic system with a single internal structure. The study does not test how models behave when coherence is partial, unstable, or conflicting.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusions","content":"\u003cp\u003eThis study examined whether LLM performance on multiple-choice tasks in a fully fictional domain reflects surface-level heuristics or coherence-based reasoning.\u003c/p\u003e \u003cp\u003eEight large language models, tested across two conditions and eighty independent sessions, achieved near-ceiling accuracy on a test built around a device with no real-world referent. Removal of the question stems did not affect performance. When reasoning was provided, models reconstructed the fictional system and generated internally consistent explanations for their answers. Human participants performed near chance.\u003c/p\u003e \u003cp\u003eThese findings indicate that, under conditions of global coherence, LLMs can construct and apply internal representations rather than relying on heuristic strategies.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch3\u003eAcknowledgements\u003c/h3\u003e\n\u003cp\u003eNo external funding was received for this work. The author is sincerely grateful to Maxime Griot, Jean Vanderdonckt, Demet Yuksel, and Coralie Hemptinne for their important contributions to the field and for the inspiration their work provided. This study would not exist without it.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eBalepur, N., Ravichander, A. \u0026amp; Rudinger, R. Artifacts or abduction: How do LLMs answer multiple-choice questions without the question? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 10308\u0026ndash;10330 (Association for Computational Linguistics, 2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBalepur, N., Rudinger, R. \u0026amp; Boyd-Graber, J. L. Which of these best describes multiple choice evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the above. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3394\u0026ndash;3418 (Association for Computational Linguistics, 2025).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCenter for AI Safety, Scale AI \u0026amp; HLE Contributors Consortium. A benchmark of expert-level academic questions to assess AI capabilities. Nature 649, 1139\u0026ndash;1146 (2026). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41586-025-09962-4\u003c/span\u003e\u003cspan address=\"10.1038/s41586-025-09962-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGriot, M., Hemptinne, C., Vanderdonckt, J. et al. Large language models lack essential metacognition for reliable medical reasoning. Nature Communications 16, 642 (2025a). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41467-024-55628-6\u003c/span\u003e\u003cspan address=\"10.1038/s41467-024-55628-6\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGriot, M., Vanderdonckt, J., Yuksel, D. \u0026amp; Hemptinne, C. Pattern recognition or medical knowledge? The problem with multiple-choice questions in medicine. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5321\u0026ndash;5341 (Association for Computational Linguistics, 2025b).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHedberg, A. Clinical reasoning in large language models under ecologically valid conditions. Zenodo (2026a). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.5281/zenodo.19600307\u003c/span\u003e\u003cspan address=\"10.5281/zenodo.19600307\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHedberg, A. The emperor\u0026rsquo;s new clothes: On LLMs and cognition. Zenodo (2026b). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.5281/zenodo.19211576\u003c/span\u003e\u003cspan address=\"10.5281/zenodo.19211576\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRaimondi, B., Pivi, F., Evangelista, D. \u0026amp; Gabbrielli, M. The CompMath-MCQ dataset: Are LLMs ready for higher-level math? arXiv:2603.03334 (2026).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSingh, S., Alyakin, A., Alber, D. A. et al. The pitfalls of multiple-choice questions in generative AI and medical education. Scientific Reports 15, 42096 (2025). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41598-025-26036-7\u003c/span\u003e\u003cspan address=\"10.1038/s41598-025-26036-7\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang, Z., Jiang, Z., Xu, L., Hao, H. \u0026amp; Wang, R. Multiple-choice questions are efficient and robust LLM evaluators. arXiv:2405.11966 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZheng, C., Zhou, H., Meng, F., Zhou, J. \u0026amp; Huang, M. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations (ICLR) (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://openreview.net/forum?id=shr9PXz7T0\u003c/span\u003e\u003cspan address=\"https://openreview.net/forum?id=shr9PXz7T0\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-9509092/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9509092/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eMultiple-choice questions (MCQs) are widely used to evaluate large language models (LLMs), but it remains unclear whether high performance reflects heuristic test-taking strategies or coherent internal reasoning. This distinction matters for both evaluation and interpretation of LLM capabilities. If performance is driven by heuristics, MCQs may overestimate model capability. If it instead reflects coherent internal representations, MCQ results may capture more substantive reasoning processes and depend critically on task design.\u003c/p\u003e \u003cp\u003eTo address this, we constructed a fully fictional medical imaging system (Nodjoli-X34) with no real-world referent and developed a 20-item MCQ test based solely on its internal logic.\u003c/p\u003e \u003cp\u003eEight LLMs were evaluated across two conditions: a standard format including questions and answer options, and an answer-only format in which question stems were removed. Across 80 independent runs, models achieved near-ceiling performance in both conditions (99.4% and 99.1% accuracy, respectively). Removal of the question stems did not reduce performance, indicating that the information required to select the correct answer was contained entirely within the structure of the answer options. When reasoning was provided (79 of 80 runs), models reconstructed a coherent internal representation of the system and applied it consistently across items. Human participants performed at chance level.\u003c/p\u003e \u003cp\u003eThese findings indicate that, under conditions of global coherence, LLMs construct and apply internal representations rather than relying on question-level heuristics. Comparison with prior work suggests that LLM reasoning strategy is adaptive: when coherence is unstable, models rely on heuristics; when coherence is the only reliable signal, they rely on it.\u003c/p\u003e","manuscriptTitle":"How can large language models answer questions about a device that does not exist?","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-29 11:01:56","doi":"10.21203/rs.3.rs-9509092/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"c10471e7-d513-42a0-ae4f-195f4d53c6d8","owner":[],"postedDate":"April 29th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Rejected","date":"2026-04-30T08:20:54+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-04-30T08:19:23+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":67088066,"name":"Physical sciences/Mathematics and computing"},{"id":67088067,"name":"Biological sciences/Psychology"},{"id":67088068,"name":"Social science/Psychology"}],"tags":[],"updatedAt":"2026-04-30T08:26:04+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-29 11:01:56","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9509092","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9509092","identity":"rs-9509092","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}