Measuring What AI Models Know and Whether They Know What They Don't Know: A Three-Run, Blind, Cross-Domain Benchmark of Six Leading Large Language Models

Measuring What AI Models Know and Whether They Know What They Don't Know: A Three-Run, Blind, Cross-Domain Benchmark of Six Leading Large Language Models | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Measuring What AI Models Know and Whether They Know What They Don't Know: A Three-Run, Blind, Cross-Domain Benchmark of Six Leading Large Language Models Kuldeep Kumar Pandit This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8937831/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Accurate self-knowledge — the capacity to know what one does not know — may be as important as raw capability in deployed artificial intelligence systems. We report a pre-registered, three-run, blinded evaluation of six frontier large language models (Claude, ChatGPT, Gemini, Grok, DeepSeek, and Perplexity) on ten cross-domain questions spanning thermodynamics, limnology, biostatistics, cardiovascular physiology, evolutionary biology, atmospheric chemistry, Bayesian statistics, evolutionary anthropology, fluid dynamics, and game theory. Each question required not only a correct conclusion but a complete multi-criterion reasoning chain; partial credit was awarded for incomplete but directionally correct reasoning. The same protocol was administered across three independent runs (n = 180 total iterations). Two outcome measures were collected: (1) independently-scored answer quality (0.0 / 0.5 / 1.0 per question per run) and (2) self-audit accuracy — the model's own verdict on its performance against disclosed ground truth. Models differed substantially in mean answer quality (range 7.17–9.17/10; Kruskal-Wallis H = 8.49, p = 0.13; Cohen's d Claude vs Perplexity = 6.93). Critically, self-audit reliability varied even more dramatically: Claude achieved near-perfect self-calibration (Spearman ρ = 0.90, mean discrepancy +0.33), while Grok exhibited extreme systematic deflation (mean discrepancy −7.5), DeepSeek showed run-to-run incoherence (discrepancies +0.5, −3.5, +2.5 across identical-quality answers), and Perplexity lapsed into hallucination for one domain across two runs. Four of ten questions achieved universal 100% success; four questions produced partial success rates of 72–83%, revealing structurally embedded knowledge gaps resistant to correction by ground-truth exposure alone. Token verbosity showed no positive correlation with accuracy (Pearson r = 0.28, p = 0.15), while Gemini achieved the highest token efficiency (6.16 accuracy points per 1,000 tokens). The results suggest that current LLMs possess stratified epistemic competencies: a reliable core of common-curriculum reasoning that is fully consistent, surrounded by domains where mechanism-level specificity remains systematically incomplete. Self-audit performance is an independent dimension of capability that diverges sharply from answer quality and constitutes a significant dimension for model evaluation. Biological sciences/Computational biology and bioinformatics Biological sciences/Evolution Physical sciences/Mathematics and computing Biological sciences/Psychology Social science/Psychology large language model evaluation self-calibration epistemic metacognition benchmark cross-domain reasoning self-audit reliability AI safety Figures Figure 1 Figure 2 1. Introduction The rapid deployment of large language models (LLMs) across domains from clinical decision support to legal analysis raises an urgent question: when these systems are wrong, do they know it? The consequences of confident errors differ fundamentally from the consequences of correctly-flagged uncertainty. A physician acting on an AI-generated diagnosis, a researcher accepting an AI-synthesised summary, or a policymaker using an AI risk assessment each depends not only on the model's accuracy but on the reliability of the model's self-reported confidence. Existing benchmarks predominantly measure answer accuracy on standardised test sets [ 1 , 2 , 3 ] . The capacity of models to accurately assess their own outputs — whether they know what they do not know — has received comparatively little systematic attention [ 10 ] . Related work on model calibration addresses probability estimation for classification tasks [ 12 ] , but does not evaluate whether models can identify which specific answers are wrong after completing open-domain expert queries. The truthfulness of model outputs has been studied via TruthfulQA [ 11 ] , but self-audit of reasoning quality under blinded, repeated conditions remains unmeasured. The distinction matters: a model that answers 8/10 questions correctly and accurately identifies which 2 it failed is qualitatively safer for deployment than one that answers 9/10 but systematically misrepresents which answer is wrong. We designed a benchmark with three novel features. First, questions were drawn from ten distinct domains to assess cross-domain reasoning rather than single-subject depth. Second, scoring required not just correct conclusions but complete multi-criterion reasoning chains, using discriminating criteria that distinguish surface-level recall from mechanistic understanding. Third, each model was asked to self-audit against disclosed ground truth after completing the test — providing a direct measure of self-audit reliability independent of answer quality. The evaluation was administered in three independent, identical runs to each of six leading models, yielding 180 total iterations. This repeated-measures design allows assessment of consistency as a proxy for reliability. 2. Methods 2.1 Design This was a pre-specified, blinded benchmark evaluation. The same 10-question protocol was administered three times to each model (runs were separated by session boundaries to prevent context carry-over). The evaluator (K.P.) was blind to model identity during scoring. Ground truth for each question was pre-specified based on established scientific literature before any model outputs were collected. 2.2 Models Six frontier LLMs were evaluated in their publicly-available configurations at the time of testing (February 2026). Exact model versions were: Claude Sonnet 4.6 (Anthropic; paid subscription); ChatGPT GPT-5.2 (OpenAI; paid subscription); Gemini 3 Flash (Google; free tier); Grok 4.2 (xAI; paid subscription); DeepSeek R1 (DeepSeek AI; free tier); and Perplexity powered by Grok 4.1 (Perplexity AI; free tier). No fine-tuning, retrieval augmentation, or custom system prompts were applied beyond the standard evaluation protocol. All queries were submitted via each model's standard consumer interface. 2.3 Questions and Ground Truth Ten questions were selected to represent distinct epistemic domains: Applied Thermodynamics (Q1), Limnology (Q2), Clinical Biostatistics (Q3), Cardiovascular Physiology (Q4), Evolutionary Entomology (Q5), Atmospheric Chemistry (Q6), Bayesian Medicine (Q7), Evolutionary Anthropology (Q8), Fluid Dynamics and Ornithology (Q9), and Game Theory applied to Ecology (Q10). Questions were designed to have a single scientifically defensible answer but to permit partial credit for answers that were directionally correct but missing specific mechanistic criteria. Each question was scored against four pre-specified criteria (A–D), where D was designated the 'key discriminating criterion' — the one most likely to separate superficial from mechanistic understanding. Examples include: Secchi disk as the early-warning instrument for Q2 (not merely 'increased turbidity'); L1014F or equivalent kdr mutation specified for Q5; anti-phase wingbeat adjustment in the downwash zone (not merely phase synchronisation to upwash) for Q9; and graduated sanctions cited as the specific mechanism in Ostrom's (1990) [ 4 ] common-pool resource framework for Q10. 2.4 Scoring Each question was scored on a three-level ordinal scale: 1.0 (all four criteria met), 0.5 (partial — directionally correct but one or more discriminating criteria absent), 0.0 (wrong or misleading). Total scores per run ranged from 0 to 10. Scoring was conducted independently by the author against pre-specified rubrics. A second rater assessed 30% of responses; inter-rater agreement was Cohen's κ = 0.87. 2.5 Self-Audit Procedure After completing all 10 questions, each model was presented with the ground truth and asked to evaluate its own performance on a question-by-question basis. Self-audit claims were encoded as 1.0 (claimed exactly right), 0.5 (claimed partially correct), or 0.0 (claimed wrong). Discrepancy scores were computed as claimed_score − actual_score at both question and run level. 2.6 Statistical Analysis Given n = 3 runs per model, the study was deliberately small in scale; formal statistical power to reject the omnibus null hypothesis was limited by design. Non-parametric tests were therefore used throughout as the most appropriate approach for ordinal data with small samples. Model differences in accuracy were assessed using Kruskal-Wallis H (omnibus) and pairwise Mann-Whitney U tests with Bonferroni correction (adjusted α = 0.003 for 15 comparisons). Because p-values are sensitive to sample size and a non-significant omnibus test does not preclude practically important differences, effect sizes (Cohen's d) and 95% bootstrap confidence intervals (10,000 resamples) are treated as the primary evidence throughout. DeepSeek's identical score across all three runs (SD = 0.00) renders Cohen's d undefined for comparisons involving that model; these are reported as 'undefined (zero variance)' rather than inflated. Token-accuracy correlations were assessed by Pearson r and Spearman ρ. Self-audit reliability was quantified as the Spearman ρ between claimed and actual question-level scores across all 30 observations per model (3 runs × 10 questions). 3. Results 3.1 Answer Accuracy: Question-Level Analysis Table 1 presents question-level accuracy pooled across all 6 models and 3 runs (n = 18 observations per question). Four questions achieved universal perfect scores (mean = 1.000): Q3 (Biostatistics), Q6 (Atmospheric Chemistry), Q7 (Bayesian Medicine), and Q8 (Evolutionary Anthropology). These represent a domain cluster we term the 'reliable core' — questions where LLMs demonstrate consistent mechanistic understanding regardless of model identity or run. In sharp contrast, four questions achieved partial credit in 83% of all iterations (15/18): Q2 (Limnology — Secchi disk not specifically cited), Q5 (Entomology — L1014F or specific CYP subfamily not cited), and Q9 (Fluid Dynamics — anti-phase wingbeat adjustment in the downwash zone not described). Q4 (Cardiovascular Physiology) failed partially in 72% of iterations (13/18), with one complete failure (0.0), because the majority of models identified collateral circulation rather than coronary autoregulation as the primary resting compensation mechanism. Table 1 Question-level accuracy and failure rates (n = 18 iterations per question). Q Domain Ground Truth Key Perfect 1.0 (of 18) Partial 0.5 (of 18) Fail 0.0 (of 18) Mean Score Failure Type Q1 Applied Physics Regular ice + both temp bounds + calc. 17/18 1/18 0/18 0.972 — Q2 Limnology Full mechanism chain + Secchi disk 3/18 15/18 0/18 0.583 Type I Q3 Biostatistics Psychiatrist correct + 7-pt threshold + n=50 18/18 0/18 0/18 1.000 — Q4 Cardiovascular Physiol. Autoregulation as PRIMARY mechanism 5/18 12/18 1/18 0.611 Type II Q5 Evolutionary Biology L1014F/kdr + CYP enzymes + formula 3/18 15/18 0/18 0.583 Type III Q6 Atmos. Chemistry Global CFCs + PSC chemistry + two-stage 18/18 0/18 0/18 1.000 — Q7 Bayesian Medicine PPV = 3.3% + base rate neglect named 18/18 0/18 0/18 1.000 — Q8 Evolutionary Anthropol. Protein denat. + starch gelat. + ETH + fossil 18/18 0/18 0/18 1.000 — Q9 Fluid Dynamics Phase sync to upwash + anti-phase downwash 3/18 15/18 0/18 0.583 Type III Q10 Game Theory/Ecology Ostrom CPR + grad. sanctions + payoff matrix 7/18 9/18 2/18 0.639 Type I The pattern of failures reveals structurally embedded knowledge gaps. For Q2, all models except Gemini correctly described the mechanism chain (phosphorus → algal bloom → oxygen depletion → fish death) but defaulted to 'green surface scum' or 'water clarity' as the early warning sign, failing to identify the Secchi disk — the canonical limnological measurement instrument. This represents Type I failure: instrument-specific nomenclature absent from generalist training distributions. For Q4, models correctly identified that compensation occurs at rest but misidentified its nature, listing collateral vessels first or without priority. This represents Type II failure: mechanism hierarchy errors correctible by ground-truth exposure. For Q5 and Q9, missing criteria (L1014F for Q5; anti-phase phase adjustment for Q9) represent Type III failure: structurally embedded gaps where generic category labels (kdr mutation, phase synchronisation) substitute for specific molecular or experimental identifiers. 3.2 Answer Accuracy: Model-Level Analysis Table 2 summarises model performance across three runs with confidence intervals and token efficiency. Claude achieved the highest mean accuracy (9.17/10, 95% CI [9.00, 9.50]) and the most consistent performance (SD = 0.29). Gemini achieved the second-highest mean (8.50/10) but showed the highest run-to-run variability (SD = 0.87), with a notable complete failure on Q4 in Run 1 (score = 0.0) that self-corrected in Runs 2 and 3. DeepSeek showed perfect consistency across runs (SD = 0.00, 7.50 in all three), suggesting deterministic or near-deterministic output generation. Table 2 Model performance across three runs with statistical summary and token efficiency. Model R1 R2 R3 Mean SD 95% CI Token Avg Pts/1000 tok Claude 9.0 9.0 9.5 9.17 0.29 [9.0, 9.5] 2833 3.236 ChatGPT 8.0 8.0 7.0 7.67 0.58 [7.0, 8.0] 2500 3.067 Gemini 7.5 9.0 9.0 8.50 0.87 [7.5, 9.0] 1393 6.102 Grok 8.0 8.0 7.5 7.83 0.29 [7.5, 8.0] 2700 2.901 DeepSeek 7.5 7.5 7.5 7.50 0.00 [7.5, 7.5] 2367 3.169 Perplexity 7.0 7.0 7.5 7.17 0.29 [7.0, 7.5] 1400 5.119 The Kruskal-Wallis omnibus test yielded H = 8.49 (p = 0.130), non-significant at conventional thresholds — a predictable consequence of the small within-group sample size (n = 3 per model), not evidence of absent differences. The primary evidence for differentiation lies in effect sizes: Claude versus Perplexity, d = 6.93 (exceptionally large); Claude versus ChatGPT, d = 3.29 (large); Gemini versus Perplexity, d = 2.07 (large). For DeepSeek, Cohen's d is undefined for all pairwise comparisons because its score was identical across all three runs (SD = 0.00), preventing pooled standard deviation calculation. Bootstrap 95% CIs for Claude ([9.00, 9.50]) and Perplexity ([7.00, 7.50]) do not overlap, providing convergent evidence of a genuine performance differential (see Fig. 1 ). 3.3 Self-Audit Accuracy Table 3 presents self-audit discrepancy data and Fig. 2 provides visual comparison of Spearman ρ values and actual versus claimed scores across all six models. This is the study's most striking finding. Self-audit quality varied enormously — from near-perfect calibration (Claude, ρ = 0.903) to complete inversion of true performance (Grok, mean discrepancy − 7.5) — see Fig. 2 . Table 3 Self-audit discrepancy: claimed versus actual scores (positive = inflation, negative = deflation). Model R1 Actual R1 Claimed R2 Actual R2 Claimed R3 Actual R3 Claimed Mean Δ Pattern rho Claude 9.0 9.0 9.0 9.5 9.5 10.0 + 0.0 / +0.5 / +0.5 Calibrated with slight inflation 0.903 ChatGPT 8.0 10.0 8.0 6.0 7.0 9.0 + 2.0 / -2.0 / +2.0 Inconsistent (bi-directional) 0.340 Gemini 7.5 10.0 9.0 10.0 9.0 10.0 + 2.5 / +1.0 / +1.0 Persistent inflation 0.658 Grok 8.0 0.0 8.0 1.0 7.5 0.0 -8.0 / -7.0 / -7.5 Extreme systematic deflation 0.396 DeepSeek 7.5 8.0 7.5 4.0 7.5 10.0 + 0.5 / -3.5 / +2.5 Incoherent (run-to-run flip) 0.330 Perplexity 7.0 8.0 7.0 1.0 7.5 1.0 + 1.0 / -6.0 / -6.5 R2-R3 extreme deflation 0.091 Claude demonstrated the highest self-audit reliability (Spearman ρ = 0.90, p < 0.001), correctly identifying its two systematic failure modes (Secchi disk for Q2; primary mechanism hierarchy for Q4) in Run 1, and maintaining mean discrepancy of + 0.33. Crucially, Claude's Run 1 self-audit recognised both errors that were actually present — a demonstration of ground-truth calibration. In Runs 2 and 3, the Q4 error persisted in the answer but was no longer flagged in the self-audit, suggesting that self-audit regression can coexist with stable answer quality. Grok exhibited extreme systematic deflation across all three runs (mean discrepancy = − 7.5): it rated essentially all its responses as wrong despite achieving actual scores of 7.5–8.0/10. This is not a conservative bias but a functional failure of self-assessment. Gemini exhibited the opposite pattern — persistent inflation (mean discrepancy = + 1.5) — claiming perfect 10/10 in all three runs despite scoring 7.5 (Run 1), 9.0 (Run 2), and 9.0 (Run 3). Grok's deflation was more consistent across runs; Gemini's inflation was more consistent across questions. DeepSeek's self-audit was uniquely incoherent: Run 1 produced reasonably calibrated self-assessment (discrepancy = + 0.5), Run 2 produced extreme deflation (− 3.5), and Run 3 produced extreme inflation (+ 2.5) — for functionally identical answer quality (all three runs scored 7.5/10). This run-to-run reversal on identical content is unprecedented in our data and suggests that DeepSeek's self-evaluation process is substantially stochastic or disconnected from actual answer quality. Perplexity exhibited a hallucination in Q10 in Runs 1 and 2: it identified 'ROSCAs (rotating savings/credit associations)' as Elinor Ostrom's institutional mechanism — a factual fabrication; Ostrom studied fisheries, forests, and irrigation commons, not credit associations. Self-audit in Runs 2 and 3 rated all questions as wrong except Q7, an extreme under-claim given actual scores of 7.0–7.5. 3.4 Token Efficiency Table 4 presents token usage and efficiency metrics. There was no significant positive correlation between token usage and accuracy (Pearson r = 0.28, p = 0.146; Spearman ρ = 0.40, p = 0.099). Gemini achieved both the second-highest mean accuracy (8.50/10) and the lowest mean token usage (1,393 tokens per run), yielding by far the highest token efficiency (6.16 accuracy points per 1,000 tokens). Claude achieved the highest accuracy but at higher token cost (2,833 tokens average), yielding 3.24 points per 1,000 tokens. Grok consumed the most tokens (2,700 average) with the second-lowest efficiency (2.91 points per 1,000 tokens). Table 4 Token usage and efficiency analysis (accuracy points per 1,000 tokens). Model R1 Tokens R2 Tokens R3 Tokens Avg Tokens Avg Score Pts/1000 Tok Rank Gemini 1480 1250 1450 1393 8.50 6.102 #1 Perplexity 1200 1200 1800 1400 7.17 5.119 #2 Claude 2800 2800 2900 2833 9.17 3.236 #3 DeepSeek 2500 2200 2400 2367 7.50 3.169 #4 ChatGPT 2300 2600 2600 2500 7.67 3.067 #5 Grok 2800 2850 2450 2700 7.83 2.901 #6 3.5 Statistical Summary Table 5 consolidates key statistical findings. Table 5 Statistical test results. Comparison Test Statistic p-value Interpretation All 6 models Kruskal-Wallis H = 8.485 p = 0.130 No global sig. (n = 3/group); large effect sizes confirm practical differences Claude vs Perplexity Mann-Whitney U U = 9.0 p = 0.050 Trend (α = 0.05); Cohen's d = 6.93 (exceptionally large effect) Claude vs ChatGPT Mann-Whitney U U = 9.0 p = 0.050 Trend (α = 0.05); Cohen's d = 3.29 (large effect) Claude vs DeepSeek Mann-Whitney U U = 9.0 p = 0.050 Trend; Cohen's d = ∞ (zero variance in DeepSeek) Gemini vs Perplexity Mann-Whitney U U = 8.5 p = 0.081 Cohen's d = 2.07 (large effect) Token vs Accuracy Spearman ρ = 0.399 p = 0.099 No significant positive correlation; more tokens ≠ higher accuracy Token vs Accuracy Pearson r = 0.276 p = 0.146 Confirms: token verbosity does not predict quality Self-audit (Claude) Spearman ρ = 0.903 p < 0.001 Near-perfect calibration — gold standard self-evaluation Self-audit (Perplexity) Spearman ρ = 0.091 p = 0.081 No significant correlation — self-evaluation functionally unreliable 4. Discussion 4.1 The Stratified Knowledge Structure of LLMs The most theoretically important finding is the structural stratification of model accuracy. Four questions — spanning Bayesian statistics, biostatistics, atmospheric chemistry, and evolutionary anthropology — achieved unanimous perfect scores across all 6 models and all 3 runs. These domains correspond to what might be termed 'canonical curriculum knowledge': material intensively represented in scientific pedagogy and widely reproduced in text corpora. Models converge not only on correct conclusions but on identical reasoning chains, citing the same experimental benchmarks (Gigerenzer & Hoffrage 1995 [ 5 ] for base-rate neglect; Aiello & Wheeler 1995 [ 6 ] for the Expensive Tissue Hypothesis; Portugal et al. 2014 [ 7 ] for V-formation aerodynamics). In contrast, the four systematically-failed question domains share a distinct property: the discriminating criterion requires recall of a specific instrument name, molecular code, or experimental observation that is unlikely to appear frequently in training text. The Secchi disk is mentioned millions of times less frequently than 'water clarity' in scientific literature; the CYP6/CYP9 cytochrome P450 subfamilies are named far less often than the generic 'P450'; 'anti-phase wingbeat adjustment in the downwash zone' is a specific observation from a single 2014 paper, not a generalisable principle. This suggests that LLM knowledge is not uniformly dense but has a high-frequency core surrounded by a long-tail periphery of instrument-specific and observation-specific knowledge. 4.2 Self-Audit as an Independent Capability Dimension The radical divergence between answer quality and self-audit quality — uncorrelated across models (Pearson r = 0.12 between mean accuracy rank and self-audit reliability rank) — establishes self-audit as an independent capability dimension. The extreme deflation observed in Grok (all answers rated wrong despite moderate-high actual quality) and the hallucination-contaminated self-audit of Perplexity are not simply overconfidence or underconfidence; they represent qualitatively different failure modes that carry distinct safety implications. Inflationary self-audit (Gemini's persistent + 1.5 to + 2.5 overclaiming) is dangerous in high-stakes deployment: users who prompt models to evaluate their own outputs receive false assurance. Deflationary self-audit (Grok's − 7.5 systematic underclaiming) wastes users' time by flagging correct responses for unnecessary human review. Incoherent self-audit (DeepSeek's run-to-run reversal) is perhaps the most problematic: it signals that self-evaluation produces random output, offering no useful signal at all. Any safety architecture relying on model self-assessment must account for these failure modes. 4.3 The Type III Knowledge Gap Problem We propose distinguishing three error types with different correctability profiles. Type I errors (instrument-specific recall failures, e.g., Secchi disk) can in principle be corrected by targeted fine-tuning on domain-specific nomenclature. Type II errors (mechanism hierarchy errors, e.g., collateral vs. autoregulation as primary mechanism) can be corrected by chain-of-thought reasoning or ground-truth exposure — as Gemini's self-correction of Q4 across runs demonstrates. Type III errors (structurally embedded specific knowledge gaps, e.g., anti-phase wingbeat adjustment in downwash) are particularly resistant: they involve recall of specific sub-findings from primary literature rather than teachable principles, and cannot be resolved by reasoning about what one already knows. 4.4 Limitations Several limitations constrain interpretation. The small sample size (n = 3 runs per model) limits statistical power; observed effect sizes are large but formal significance thresholds were not universally met. The benchmark comprised only 10 questions; a larger item pool would improve reliability estimates. Questions were not independently validated against an established psychometric framework, though the pre-specification of scoring criteria before data collection guards against post-hoc scoring bias. The benchmark reflects English-language capabilities only. Scoring was primarily conducted by the first author; inter-rater agreement assessed on 30% of responses (κ = 0.87) mitigates but does not eliminate this concern. 5. Conclusions This study provides the first systematic comparison of answer accuracy and self-audit reliability in six frontier LLMs across ten scientific domains using a repeated-measures blinded design. Three findings are of particular significance: Knowledge stratification is real and structured: LLMs share a reliable core of common-curriculum reasoning (evidenced by four questions achieving 100% perfect scores across all models and all runs) but diverge sharply on domain-specific nomenclature and sub-finding recall — a profile consistent with high-frequency training corpus saturation and long-tail gaps. Self-audit is an independent capability dimension: answer quality and self-audit reliability are uncorrelated across models. Models that score highly on accuracy do not necessarily score well on self-calibration, and vice versa. Self-audit quality is thus a necessary evaluation target in its own right. The architecture of AI evaluation itself needs reconsideration: benchmarks measuring answer accuracy alone are insufficient. Given the deployment of LLMs in roles where users may delegate verification to the model itself, measuring self-audit reliability — how accurately a model identifies which of its own answers are wrong — is as important as measuring what the model knows. Future work should extend this paradigm to larger question pools, additional languages, domain-expert scoring panels, and longitudinal tracking of model versions. Declarations Conflicts of interest The author declares no competing interests, financial or non-financial. No author has received funding, gifts-in-kind, honoraria, or any other consideration from any of the AI vendors whose models were evaluated in this study (Anthropic, OpenAI, Google DeepMind, xAI, DeepSeek AI, Perplexity AI). The benchmark was designed and administered without commercial sponsorship. Ethics approval This study did not involve human participants, patient data, animal subjects, or any personal data. It consisted exclusively of queries submitted to publicly available commercial LLM interfaces. Formal ethical approval was not required. Funding This work received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. All computational access was via commercially available subscription plans (paid or free tier, as specified in Section 2.2). Author contributions Kuldeep Kumar Pandit designed the benchmark, formulated the ten cross-domain questions and four-criterion scoring rubric, administered all three runs under blinded conditions, scored all model outputs, conducted the statistical analysis, and drafted the manuscript in its entirety. Data availability The full text of all model responses (180 iterations), the complete scoring rubric with criterion-level justifications, the raw scoring matrix, and all statistical code are reported in the supplementary materials accompanying this submission. No proprietary or non-public data were used. All queries were submitted to publicly available consumer interfaces; no API access or proprietary data pipelines were used. Generative AI use disclosure The models evaluated in this study (Claude Sonnet 4.6, ChatGPT-5.2, Gemini 3 Flash, Grok 4.2, DeepSeek R1, and Perplexity powered by Grok 4.1) are themselves the subject of investigation. Their outputs constitute the raw data of the study. No generative AI was used to write, edit, or revise the manuscript text. References Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2021). Measuring massive multitask language understanding. arXiv:2009.03300. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., … & Leskovec, J. (2022). Holistic evaluation of language models. arXiv:2211.09110. Srivastava, A., Rastogi, A., Rao, A., Shoaib, A. O. M., Torralba, A., Hashimoto, T., … & Wu, T. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615. Ostrom, E. (1990). Governing the Commons: The Evolution of Institutions for Collective Action. Cambridge University Press. Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684–704. Aiello, L. C., & Wheeler, P. (1995). The Expensive-Tissue Hypothesis: The brain and the digestive system in human and primate evolution. Current Anthropology, 36(2), 199–221. Portugal, S. J., Hubel, T. Y., Fritz, J., Heese, S., Trobe, D., Voelkl, B., … & Usherwood, J. R. (2014). Upwash exploitation and downwash avoidance by flap phasing in ibis formation flight. Nature, 505, 399–402. Eddy, D. M. (1982). Probabilistic reasoning in clinical medicine: Problems and opportunities. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment Under Uncertainty: Heuristics and Biases (pp. 249–267). Cambridge University Press. Hardin, G. (1968). The tragedy of the commons. Science, 162(3859), 1243–1248. Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., … & Kaplan, J. (2022). Language models (mostly) know what they know. arXiv:2207.05221. Lin, S., Hilton, J., & Evans, O. (2022). TruthfulQA: Measuring how models mimic human falsehoods. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 3214–3252. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning (ICML), 1321–1330. Additional Declarations No competing interests reported. Supplementary Files DatasheetsAnalysisStatisticsFinalnpjdigitalmedcine.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8937831","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":605360404,"identity":"751a4e01-fa2f-49d3-a46b-11bc89664c5d","order_by":0,"name":"Kuldeep Kumar Pandit","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABAUlEQVRIiWNgGAWjYBACCQbmBgko2+AAA48NkGZsPIBfCyOSlgM8aSAtDcRrYTjAcBjMwqtFckZi440PFXfy+Gc3bzz8Qea83dr2w0BbamyicWmRlkhstpxx5lmxxJ1jBUCH3U7ediYRqOVYWm4DDi1yEolt0rxthxMbbuQYgLWYHQBqYWw4jF/L33+HE+dDtJxLNjv/EL8WaZAWoILEDRAtB+zMbhCwRbLnYbNlz7FnxYY30goOnOFJTjC7AbQlAY9fJI4nH7zxo+ZOntyN5M0fKnvs7M3Opz988KHGBqcWKDiQAKYYexgSwSoT8CtH0sLwg8GesOJRMApGwSgYaQAA6gxyDFC/BUkAAAAASUVORK5CYII=","orcid":"","institution":"","correspondingAuthor":true,"prefix":"","firstName":"Kuldeep","middleName":"Kumar","lastName":"Pandit","suffix":""}],"badges":[],"createdAt":"2026-02-22 08:23:25","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8937831/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8937831/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":104681406,"identity":"0340005e-49d0-4dd7-a155-11035b965496","added_by":"auto","created_at":"2026-03-16 02:41:09","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":87328,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-8937831/v1/eeeb111f72624e916f228af2.png"},{"id":104681409,"identity":"56528b20-255a-46ad-a17b-5910c060bbf2","added_by":"auto","created_at":"2026-03-16 02:41:10","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":126685,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-8937831/v1/d3e2364d8f61d69f3639ecaf.png"},{"id":104782238,"identity":"61165039-4fc3-4658-8910-8a13fdc1dab4","added_by":"auto","created_at":"2026-03-17 07:57:01","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1165907,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8937831/v1/e297768d-b73b-4844-9a8e-e07bb2b254f6.pdf"},{"id":104681407,"identity":"b96ff565-7359-4ac1-8298-26e916b2b8e3","added_by":"auto","created_at":"2026-03-16 02:41:09","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":3363965,"visible":true,"origin":"","legend":"","description":"","filename":"DatasheetsAnalysisStatisticsFinalnpjdigitalmedcine.docx","url":"https://assets-eu.researchsquare.com/files/rs-8937831/v1/b8f15637562fd3fd3073a5d6.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Measuring What AI Models Know and Whether They Know What They Don't Know: A Three-Run, Blind, Cross-Domain Benchmark of Six Leading Large Language Models","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eThe rapid deployment of large language models (LLMs) across domains from clinical decision support to legal analysis raises an urgent question: when these systems are wrong, do they know it? The consequences of confident errors differ fundamentally from the consequences of correctly-flagged uncertainty. A physician acting on an AI-generated diagnosis, a researcher accepting an AI-synthesised summary, or a policymaker using an AI risk assessment each depends not only on the model's accuracy but on the reliability of the model's self-reported confidence.\u003c/p\u003e \u003cp\u003eExisting benchmarks predominantly measure answer accuracy on standardised test sets\u003csup\u003e[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]\u003c/sup\u003e. The capacity of models to accurately assess their own outputs \u0026mdash; whether they know what they do not know \u0026mdash; has received comparatively little systematic attention\u003csup\u003e[\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]\u003c/sup\u003e. Related work on model calibration addresses probability estimation for classification tasks\u003csup\u003e[\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]\u003c/sup\u003e, but does not evaluate whether models can identify which specific answers are wrong after completing open-domain expert queries. The truthfulness of model outputs has been studied via TruthfulQA\u003csup\u003e[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]\u003c/sup\u003e, but self-audit of reasoning quality under blinded, repeated conditions remains unmeasured. The distinction matters: a model that answers 8/10 questions correctly and accurately identifies which 2 it failed is qualitatively safer for deployment than one that answers 9/10 but systematically misrepresents which answer is wrong.\u003c/p\u003e \u003cp\u003eWe designed a benchmark with three novel features. First, questions were drawn from ten distinct domains to assess cross-domain reasoning rather than single-subject depth. Second, scoring required not just correct conclusions but complete multi-criterion reasoning chains, using discriminating criteria that distinguish surface-level recall from mechanistic understanding. Third, each model was asked to self-audit against disclosed ground truth after completing the test \u0026mdash; providing a direct measure of self-audit reliability independent of answer quality.\u003c/p\u003e \u003cp\u003eThe evaluation was administered in three independent, identical runs to each of six leading models, yielding 180 total iterations. This repeated-measures design allows assessment of consistency as a proxy for reliability.\u003c/p\u003e"},{"header":"2. Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Design\u003c/h2\u003e \u003cp\u003eThis was a pre-specified, blinded benchmark evaluation. The same 10-question protocol was administered three times to each model (runs were separated by session boundaries to prevent context carry-over). The evaluator (K.P.) was blind to model identity during scoring. Ground truth for each question was pre-specified based on established scientific literature before any model outputs were collected.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Models\u003c/h2\u003e \u003cp\u003eSix frontier LLMs were evaluated in their publicly-available configurations at the time of testing (February 2026). Exact model versions were: Claude Sonnet 4.6 (Anthropic; paid subscription); ChatGPT GPT-5.2 (OpenAI; paid subscription); Gemini 3 Flash (Google; free tier); Grok 4.2 (xAI; paid subscription); DeepSeek R1 (DeepSeek AI; free tier); and Perplexity powered by Grok 4.1 (Perplexity AI; free tier). No fine-tuning, retrieval augmentation, or custom system prompts were applied beyond the standard evaluation protocol. All queries were submitted via each model's standard consumer interface.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Questions and Ground Truth\u003c/h2\u003e \u003cp\u003eTen questions were selected to represent distinct epistemic domains: Applied Thermodynamics (Q1), Limnology (Q2), Clinical Biostatistics (Q3), Cardiovascular Physiology (Q4), Evolutionary Entomology (Q5), Atmospheric Chemistry (Q6), Bayesian Medicine (Q7), Evolutionary Anthropology (Q8), Fluid Dynamics and Ornithology (Q9), and Game Theory applied to Ecology (Q10). Questions were designed to have a single scientifically defensible answer but to permit partial credit for answers that were directionally correct but missing specific mechanistic criteria.\u003c/p\u003e \u003cp\u003eEach question was scored against four pre-specified criteria (A\u0026ndash;D), where D was designated the 'key discriminating criterion' \u0026mdash; the one most likely to separate superficial from mechanistic understanding. Examples include: Secchi disk as the early-warning instrument for Q2 (not merely 'increased turbidity'); L1014F or equivalent kdr mutation specified for Q5; anti-phase wingbeat adjustment in the downwash zone (not merely phase synchronisation to upwash) for Q9; and graduated sanctions cited as the specific mechanism in Ostrom's (1990)\u003csup\u003e[\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]\u003c/sup\u003e common-pool resource framework for Q10.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e2.4 Scoring\u003c/h2\u003e \u003cp\u003eEach question was scored on a three-level ordinal scale: 1.0 (all four criteria met), 0.5 (partial \u0026mdash; directionally correct but one or more discriminating criteria absent), 0.0 (wrong or misleading). Total scores per run ranged from 0 to 10. Scoring was conducted independently by the author against pre-specified rubrics. A second rater assessed 30% of responses; inter-rater agreement was Cohen's κ\u0026thinsp;=\u0026thinsp;0.87.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e2.5 Self-Audit Procedure\u003c/h2\u003e \u003cp\u003eAfter completing all 10 questions, each model was presented with the ground truth and asked to evaluate its own performance on a question-by-question basis. Self-audit claims were encoded as 1.0 (claimed exactly right), 0.5 (claimed partially correct), or 0.0 (claimed wrong). Discrepancy scores were computed as claimed_score\u0026thinsp;\u0026minus;\u0026thinsp;actual_score at both question and run level.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e2.6 Statistical Analysis\u003c/h2\u003e \u003cp\u003eGiven n\u0026thinsp;=\u0026thinsp;3 runs per model, the study was deliberately small in scale; formal statistical power to reject the omnibus null hypothesis was limited by design. Non-parametric tests were therefore used throughout as the most appropriate approach for ordinal data with small samples. Model differences in accuracy were assessed using Kruskal-Wallis H (omnibus) and pairwise Mann-Whitney U tests with Bonferroni correction (adjusted α\u0026thinsp;=\u0026thinsp;0.003 for 15 comparisons). Because p-values are sensitive to sample size and a non-significant omnibus test does not preclude practically important differences, effect sizes (Cohen's d) and 95% bootstrap confidence intervals (10,000 resamples) are treated as the primary evidence throughout. DeepSeek's identical score across all three runs (SD\u0026thinsp;=\u0026thinsp;0.00) renders Cohen's d undefined for comparisons involving that model; these are reported as 'undefined (zero variance)' rather than inflated. Token-accuracy correlations were assessed by Pearson r and Spearman ρ. Self-audit reliability was quantified as the Spearman ρ between claimed and actual question-level scores across all 30 observations per model (3 runs \u0026times; 10 questions).\u003c/p\u003e \u003c/div\u003e"},{"header":"3. Results","content":"\u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Answer Accuracy: Question-Level Analysis\u003c/h2\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e presents question-level accuracy pooled across all 6 models and 3 runs (n\u0026thinsp;=\u0026thinsp;18 observations per question). Four questions achieved universal perfect scores (mean\u0026thinsp;=\u0026thinsp;1.000): Q3 (Biostatistics), Q6 (Atmospheric Chemistry), Q7 (Bayesian Medicine), and Q8 (Evolutionary Anthropology). These represent a domain cluster we term the 'reliable core' \u0026mdash; questions where LLMs demonstrate consistent mechanistic understanding regardless of model identity or run.\u003c/p\u003e \u003cp\u003eIn sharp contrast, four questions achieved partial credit in 83% of all iterations (15/18): Q2 (Limnology \u0026mdash; Secchi disk not specifically cited), Q5 (Entomology \u0026mdash; L1014F or specific CYP subfamily not cited), and Q9 (Fluid Dynamics \u0026mdash; anti-phase wingbeat adjustment in the downwash zone not described). Q4 (Cardiovascular Physiology) failed partially in 72% of iterations (13/18), with one complete failure (0.0), because the majority of models identified collateral circulation rather than coronary autoregulation as the primary resting compensation mechanism.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eQuestion-level accuracy and failure rates (n\u0026thinsp;=\u0026thinsp;18 iterations per question).\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"8\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003e Q\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDomain\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGround Truth Key\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePerfect 1.0 (of 18)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003ePartial 0.5 (of 18)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eFail 0.0 (of 18)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eMean Score\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eFailure Type\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eQ1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eApplied Physics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eRegular ice\u0026thinsp;+\u0026thinsp;both temp bounds\u0026thinsp;+\u0026thinsp;calc.\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e17/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.972\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u0026mdash;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eQ2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLimnology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eFull mechanism chain\u0026thinsp;+\u0026thinsp;Secchi disk\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e15/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.583\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003eType I\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eQ3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBiostatistics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003ePsychiatrist correct\u0026thinsp;+\u0026thinsp;7-pt threshold\u0026thinsp;+\u0026thinsp;n=50\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e18/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e1.000\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u0026mdash;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eQ4\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCardiovascular Physiol.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eAutoregulation as PRIMARY mechanism\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e5/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e12/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.611\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003eType II\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eQ5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEvolutionary Biology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eL1014F/kdr\u0026thinsp;+\u0026thinsp;CYP enzymes\u0026thinsp;+\u0026thinsp;formula\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e15/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.583\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003eType III\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eQ6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAtmos. Chemistry\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eGlobal CFCs\u0026thinsp;+\u0026thinsp;PSC chemistry\u0026thinsp;+\u0026thinsp;two-stage\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e18/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e1.000\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u0026mdash;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eQ7\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBayesian Medicine\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003ePPV\u0026thinsp;=\u0026thinsp;3.3% + base rate neglect named\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e18/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e1.000\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u0026mdash;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eQ8\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEvolutionary Anthropol.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eProtein denat. + starch gelat. + ETH\u0026thinsp;+\u0026thinsp;fossil\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e18/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e1.000\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u0026mdash;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eQ9\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFluid Dynamics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003ePhase sync to upwash\u0026thinsp;+\u0026thinsp;anti-phase downwash\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e15/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.583\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003eType III\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eQ10\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGame Theory/Ecology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eOstrom CPR\u0026thinsp;+\u0026thinsp;grad. sanctions\u0026thinsp;+\u0026thinsp;payoff matrix\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e7/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e9/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e2/18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.639\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003eType I\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe pattern of failures reveals structurally embedded knowledge gaps. For Q2, all models except Gemini correctly described the mechanism chain (phosphorus \u0026rarr; algal bloom \u0026rarr; oxygen depletion \u0026rarr; fish death) but defaulted to 'green surface scum' or 'water clarity' as the early warning sign, failing to identify the Secchi disk \u0026mdash; the canonical limnological measurement instrument. This represents Type I failure: instrument-specific nomenclature absent from generalist training distributions. For Q4, models correctly identified that compensation occurs at rest but misidentified its nature, listing collateral vessels first or without priority. This represents Type II failure: mechanism hierarchy errors correctible by ground-truth exposure. For Q5 and Q9, missing criteria (L1014F for Q5; anti-phase phase adjustment for Q9) represent Type III failure: structurally embedded gaps where generic category labels (kdr mutation, phase synchronisation) substitute for specific molecular or experimental identifiers.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Answer Accuracy: Model-Level Analysis\u003c/h2\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e summarises model performance across three runs with confidence intervals and token efficiency. Claude achieved the highest mean accuracy (9.17/10, 95% CI [9.00, 9.50]) and the most consistent performance (SD\u0026thinsp;=\u0026thinsp;0.29). Gemini achieved the second-highest mean (8.50/10) but showed the highest run-to-run variability (SD\u0026thinsp;=\u0026thinsp;0.87), with a notable complete failure on Q4 in Run 1 (score\u0026thinsp;=\u0026thinsp;0.0) that self-corrected in Runs 2 and 3. DeepSeek showed perfect consistency across runs (SD\u0026thinsp;=\u0026thinsp;0.00, 7.50 in all three), suggesting deterministic or near-deterministic output generation.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eModel performance across three runs with statistical summary and token efficiency.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"9\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eR1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eR2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eR3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMean\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eSD\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003e95% CI\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eToken Avg\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c9\"\u003e \u003cp\u003ePts/1000 tok\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eClaude\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e9.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e9.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e9.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e9.17\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e[9.0, 9.5]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e2833\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e3.236\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eChatGPT\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e8.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e8.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e7.67\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.58\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e[7.0, 8.0]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e2500\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e3.067\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGemini\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e9.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e9.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e8.50\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e[7.5, 9.0]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e1393\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e6.102\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGrok\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e8.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e8.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e7.83\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e[7.5, 8.0]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e2700\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e2.901\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eDeepSeek\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e7.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e7.50\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e[7.5, 7.5]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e2367\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e3.169\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ePerplexity\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e7.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e7.17\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e[7.0, 7.5]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e1400\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e5.119\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe Kruskal-Wallis omnibus test yielded H\u0026thinsp;=\u0026thinsp;8.49 (p\u0026thinsp;=\u0026thinsp;0.130), non-significant at conventional thresholds \u0026mdash; a predictable consequence of the small within-group sample size (n\u0026thinsp;=\u0026thinsp;3 per model), not evidence of absent differences. The primary evidence for differentiation lies in effect sizes: Claude versus Perplexity, d\u0026thinsp;=\u0026thinsp;6.93 (exceptionally large); Claude versus ChatGPT, d\u0026thinsp;=\u0026thinsp;3.29 (large); Gemini versus Perplexity, d\u0026thinsp;=\u0026thinsp;2.07 (large). For DeepSeek, Cohen's d is undefined for all pairwise comparisons because its score was identical across all three runs (SD\u0026thinsp;=\u0026thinsp;0.00), preventing pooled standard deviation calculation. Bootstrap 95% CIs for Claude ([9.00, 9.50]) and Perplexity ([7.00, 7.50]) do not overlap, providing convergent evidence of a genuine performance differential (see Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Self-Audit Accuracy\u003c/h2\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e presents self-audit discrepancy data and Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e provides visual comparison of Spearman ρ values and actual versus claimed scores across all six models. This is the study's most striking finding. Self-audit quality varied enormously \u0026mdash; from near-perfect calibration (Claude, ρ\u0026thinsp;=\u0026thinsp;0.903) to complete inversion of true performance (Grok, mean discrepancy\u0026thinsp;\u0026minus;\u0026thinsp;7.5) \u0026mdash; see Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSelf-audit discrepancy: claimed versus actual scores (positive\u0026thinsp;=\u0026thinsp;inflation, negative\u0026thinsp;=\u0026thinsp;deflation).\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"10\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eR1 Actual\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eR1 Claimed\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eR2 Actual\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eR2 Claimed\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eR3 Actual\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eR3 Claimed\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eMean Δ\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c9\"\u003e \u003cp\u003ePattern\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c10\"\u003e \u003cp\u003erho\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eClaude\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e9.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e9.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e9.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e9.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e9.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e10.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e+\u0026thinsp;0.0 / +0.5 / +0.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003eCalibrated with slight inflation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e0.903\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eChatGPT\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e8.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e10.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e8.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e6.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e7.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e9.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e+\u0026thinsp;2.0 / -2.0 / +2.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003eInconsistent (bi-directional)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e0.340\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGemini\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e10.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e9.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e10.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e9.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e10.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e+\u0026thinsp;2.5 / +1.0 / +1.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003ePersistent inflation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e0.658\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGrok\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e8.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e8.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e7.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e-8.0 / -7.0 / -7.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003eExtreme systematic deflation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e0.396\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eDeepSeek\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e8.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e4.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e7.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e10.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e+\u0026thinsp;0.5 / -3.5 / +2.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003eIncoherent (run-to-run flip)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e0.330\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ePerplexity\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e8.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e7.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e1.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e+\u0026thinsp;1.0 / -6.0 / -6.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003eR2-R3 extreme deflation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e0.091\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eClaude demonstrated the highest self-audit reliability (Spearman ρ\u0026thinsp;=\u0026thinsp;0.90, p\u0026thinsp;\u0026lt;\u0026thinsp;0.001), correctly identifying its two systematic failure modes (Secchi disk for Q2; primary mechanism hierarchy for Q4) in Run 1, and maintaining mean discrepancy of +\u0026thinsp;0.33. Crucially, Claude's Run 1 self-audit recognised both errors that were actually present \u0026mdash; a demonstration of ground-truth calibration. In Runs 2 and 3, the Q4 error persisted in the answer but was no longer flagged in the self-audit, suggesting that self-audit regression can coexist with stable answer quality.\u003c/p\u003e \u003cp\u003eGrok exhibited extreme systematic deflation across all three runs (mean discrepancy\u0026thinsp;=\u0026thinsp;\u0026minus;\u0026thinsp;7.5): it rated essentially all its responses as wrong despite achieving actual scores of 7.5\u0026ndash;8.0/10. This is not a conservative bias but a functional failure of self-assessment. Gemini exhibited the opposite pattern \u0026mdash; persistent inflation (mean discrepancy\u0026thinsp;=\u0026thinsp;+\u0026thinsp;1.5) \u0026mdash; claiming perfect 10/10 in all three runs despite scoring 7.5 (Run 1), 9.0 (Run 2), and 9.0 (Run 3). Grok's deflation was more consistent across runs; Gemini's inflation was more consistent across questions.\u003c/p\u003e \u003cp\u003eDeepSeek's self-audit was uniquely incoherent: Run 1 produced reasonably calibrated self-assessment (discrepancy\u0026thinsp;=\u0026thinsp;+\u0026thinsp;0.5), Run 2 produced extreme deflation (\u0026minus;\u0026thinsp;3.5), and Run 3 produced extreme inflation (+\u0026thinsp;2.5) \u0026mdash; for functionally identical answer quality (all three runs scored 7.5/10). This run-to-run reversal on identical content is unprecedented in our data and suggests that DeepSeek's self-evaluation process is substantially stochastic or disconnected from actual answer quality.\u003c/p\u003e \u003cp\u003ePerplexity exhibited a hallucination in Q10 in Runs 1 and 2: it identified 'ROSCAs (rotating savings/credit associations)' as Elinor Ostrom's institutional mechanism \u0026mdash; a factual fabrication; Ostrom studied fisheries, forests, and irrigation commons, not credit associations. Self-audit in Runs 2 and 3 rated all questions as wrong except Q7, an extreme under-claim given actual scores of 7.0\u0026ndash;7.5.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e3.4 Token Efficiency\u003c/h2\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e presents token usage and efficiency metrics. There was no significant positive correlation between token usage and accuracy (Pearson r\u0026thinsp;=\u0026thinsp;0.28, p\u0026thinsp;=\u0026thinsp;0.146; Spearman ρ\u0026thinsp;=\u0026thinsp;0.40, p\u0026thinsp;=\u0026thinsp;0.099). Gemini achieved both the second-highest mean accuracy (8.50/10) and the lowest mean token usage (1,393 tokens per run), yielding by far the highest token efficiency (6.16 accuracy points per 1,000 tokens). Claude achieved the highest accuracy but at higher token cost (2,833 tokens average), yielding 3.24 points per 1,000 tokens. Grok consumed the most tokens (2,700 average) with the second-lowest efficiency (2.91 points per 1,000 tokens).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eToken usage and efficiency analysis (accuracy points per 1,000 tokens).\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"8\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eR1 Tokens\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eR2 Tokens\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eR3 Tokens\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eAvg Tokens\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAvg Score\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003ePts/1000 Tok\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eRank\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGemini\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1480\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e1250\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1450\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1393\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e8.50\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e6.102\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e#1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ePerplexity\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1200\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e1200\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1800\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1400\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e7.17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e5.119\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e#2\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eClaude\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2800\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e2800\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2900\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e2833\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e9.17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e3.236\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e#3\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eDeepSeek\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2500\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e2200\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2400\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e2367\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e7.50\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e3.169\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e#4\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eChatGPT\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2300\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e2600\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2600\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e2500\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e7.67\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e3.067\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e#5\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGrok\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2800\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e2850\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2450\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e2700\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e7.83\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e2.901\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e#6\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003e3.5 Statistical Summary\u003c/h2\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e consolidates key statistical findings.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eStatistical test results.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eComparison\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTest\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eStatistic\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003ep-value\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eInterpretation\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAll 6 models\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eKruskal-Wallis\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eH\u0026thinsp;=\u0026thinsp;8.485\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003ep\u0026thinsp;=\u0026thinsp;0.130\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eNo global sig. (n\u0026thinsp;=\u0026thinsp;3/group); large effect sizes confirm practical differences\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eClaude vs Perplexity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMann-Whitney U\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eU\u0026thinsp;=\u0026thinsp;9.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003ep\u0026thinsp;=\u0026thinsp;0.050\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eTrend (α\u0026thinsp;=\u0026thinsp;0.05); Cohen's d\u0026thinsp;=\u0026thinsp;6.93 (exceptionally large effect)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eClaude vs ChatGPT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMann-Whitney U\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eU\u0026thinsp;=\u0026thinsp;9.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003ep\u0026thinsp;=\u0026thinsp;0.050\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eTrend (α\u0026thinsp;=\u0026thinsp;0.05); Cohen's d\u0026thinsp;=\u0026thinsp;3.29 (large effect)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eClaude vs DeepSeek\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMann-Whitney U\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eU\u0026thinsp;=\u0026thinsp;9.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003ep\u0026thinsp;=\u0026thinsp;0.050\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eTrend; Cohen's d = \u0026infin; (zero variance in DeepSeek)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGemini vs Perplexity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMann-Whitney U\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eU\u0026thinsp;=\u0026thinsp;8.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003ep\u0026thinsp;=\u0026thinsp;0.081\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eCohen's d\u0026thinsp;=\u0026thinsp;2.07 (large effect)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eToken vs Accuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSpearman\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eρ\u0026thinsp;=\u0026thinsp;0.399\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003ep\u0026thinsp;=\u0026thinsp;0.099\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eNo significant positive correlation; more tokens\u0026thinsp;\u0026ne;\u0026thinsp;higher accuracy\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eToken vs Accuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePearson\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003er\u0026thinsp;=\u0026thinsp;0.276\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003ep\u0026thinsp;=\u0026thinsp;0.146\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eConfirms: token verbosity does not predict quality\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSelf-audit (Claude)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSpearman\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eρ\u0026thinsp;=\u0026thinsp;0.903\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003ep\u0026thinsp;\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eNear-perfect calibration \u0026mdash; gold standard self-evaluation\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSelf-audit (Perplexity)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSpearman\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eρ\u0026thinsp;=\u0026thinsp;0.091\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003ep\u0026thinsp;=\u0026thinsp;0.081\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eNo significant correlation \u0026mdash; self-evaluation functionally unreliable\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"4. Discussion","content":"\u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003e4.1 The Stratified Knowledge Structure of LLMs\u003c/h2\u003e \u003cp\u003eThe most theoretically important finding is the structural stratification of model accuracy. Four questions \u0026mdash; spanning Bayesian statistics, biostatistics, atmospheric chemistry, and evolutionary anthropology \u0026mdash; achieved unanimous perfect scores across all 6 models and all 3 runs. These domains correspond to what might be termed 'canonical curriculum knowledge': material intensively represented in scientific pedagogy and widely reproduced in text corpora. Models converge not only on correct conclusions but on identical reasoning chains, citing the same experimental benchmarks (Gigerenzer \u0026amp; Hoffrage 1995\u003csup\u003e[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]\u003c/sup\u003e for base-rate neglect; Aiello \u0026amp; Wheeler 1995\u003csup\u003e[\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]\u003c/sup\u003e for the Expensive Tissue Hypothesis; Portugal et al. 2014\u003csup\u003e[\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]\u003c/sup\u003e for V-formation aerodynamics).\u003c/p\u003e \u003cp\u003eIn contrast, the four systematically-failed question domains share a distinct property: the discriminating criterion requires recall of a specific instrument name, molecular code, or experimental observation that is unlikely to appear frequently in training text. The Secchi disk is mentioned millions of times less frequently than 'water clarity' in scientific literature; the CYP6/CYP9 cytochrome P450 subfamilies are named far less often than the generic 'P450'; 'anti-phase wingbeat adjustment in the downwash zone' is a specific observation from a single 2014 paper, not a generalisable principle. This suggests that LLM knowledge is not uniformly dense but has a high-frequency core surrounded by a long-tail periphery of instrument-specific and observation-specific knowledge.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Self-Audit as an Independent Capability Dimension\u003c/h2\u003e \u003cp\u003eThe radical divergence between answer quality and self-audit quality \u0026mdash; uncorrelated across models (Pearson r\u0026thinsp;=\u0026thinsp;0.12 between mean accuracy rank and self-audit reliability rank) \u0026mdash; establishes self-audit as an independent capability dimension. The extreme deflation observed in Grok (all answers rated wrong despite moderate-high actual quality) and the hallucination-contaminated self-audit of Perplexity are not simply overconfidence or underconfidence; they represent qualitatively different failure modes that carry distinct safety implications.\u003c/p\u003e \u003cp\u003eInflationary self-audit (Gemini's persistent\u0026thinsp;+\u0026thinsp;1.5 to +\u0026thinsp;2.5 overclaiming) is dangerous in high-stakes deployment: users who prompt models to evaluate their own outputs receive false assurance. Deflationary self-audit (Grok's\u0026thinsp;\u0026minus;\u0026thinsp;7.5 systematic underclaiming) wastes users' time by flagging correct responses for unnecessary human review. Incoherent self-audit (DeepSeek's run-to-run reversal) is perhaps the most problematic: it signals that self-evaluation produces random output, offering no useful signal at all. Any safety architecture relying on model self-assessment must account for these failure modes.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003e4.3 The Type III Knowledge Gap Problem\u003c/h2\u003e \u003cp\u003eWe propose distinguishing three error types with different correctability profiles. Type I errors (instrument-specific recall failures, e.g., Secchi disk) can in principle be corrected by targeted fine-tuning on domain-specific nomenclature. Type II errors (mechanism hierarchy errors, e.g., collateral vs. autoregulation as primary mechanism) can be corrected by chain-of-thought reasoning or ground-truth exposure \u0026mdash; as Gemini's self-correction of Q4 across runs demonstrates. Type III errors (structurally embedded specific knowledge gaps, e.g., anti-phase wingbeat adjustment in downwash) are particularly resistant: they involve recall of specific sub-findings from primary literature rather than teachable principles, and cannot be resolved by reasoning about what one already knows.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec19\" class=\"Section2\"\u003e \u003ch2\u003e4.4 Limitations\u003c/h2\u003e \u003cp\u003eSeveral limitations constrain interpretation. The small sample size (n\u0026thinsp;=\u0026thinsp;3 runs per model) limits statistical power; observed effect sizes are large but formal significance thresholds were not universally met. The benchmark comprised only 10 questions; a larger item pool would improve reliability estimates. Questions were not independently validated against an established psychometric framework, though the pre-specification of scoring criteria before data collection guards against post-hoc scoring bias. The benchmark reflects English-language capabilities only. Scoring was primarily conducted by the first author; inter-rater agreement assessed on 30% of responses (κ\u0026thinsp;=\u0026thinsp;0.87) mitigates but does not eliminate this concern.\u003c/p\u003e \u003c/div\u003e"},{"header":"5. Conclusions","content":"\u003cp\u003eThis study provides the first systematic comparison of answer accuracy and self-audit reliability in six frontier LLMs across ten scientific domains using a repeated-measures blinded design. Three findings are of particular significance:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eKnowledge stratification is real and structured: LLMs share a reliable core of common-curriculum reasoning (evidenced by four questions achieving 100% perfect scores across all models and all runs) but diverge sharply on domain-specific nomenclature and sub-finding recall \u0026mdash; a profile consistent with high-frequency training corpus saturation and long-tail gaps.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eSelf-audit is an independent capability dimension: answer quality and self-audit reliability are uncorrelated across models. Models that score highly on accuracy do not necessarily score well on self-calibration, and vice versa. Self-audit quality is thus a necessary evaluation target in its own right.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eThe architecture of AI evaluation itself needs reconsideration: benchmarks measuring answer accuracy alone are insufficient. Given the deployment of LLMs in roles where users may delegate verification to the model itself, measuring self-audit reliability \u0026mdash; how accurately a model identifies which of its own answers are wrong \u0026mdash; is as important as measuring what the model knows.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eFuture work should extend this paradigm to larger question pools, additional languages, domain-expert scoring panels, and longitudinal tracking of model versions.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eConflicts of interest\u003c/p\u003e\n\u003cp\u003eThe author declares no competing interests, financial or non-financial. No author has received funding, gifts-in-kind, honoraria, or any other consideration from any of the AI vendors whose models were evaluated in this study (Anthropic, OpenAI, Google DeepMind, xAI, DeepSeek AI, Perplexity AI). The benchmark was designed and administered without commercial sponsorship.\u003c/p\u003e\n\u003cp\u003eEthics approval\u003c/p\u003e\n\u003cp\u003eThis study did not involve human participants, patient data, animal subjects, or any personal data. It consisted exclusively of \u0026nbsp;queries submitted to publicly available commercial LLM interfaces. Formal ethical approval was not required.\u003c/p\u003e\n\u003cp\u003eFunding\u003c/p\u003e\n\u003cp\u003eThis work received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. All computational access was via commercially available subscription plans (paid or free tier, as specified in Section 2.2).\u003c/p\u003e\n\u003cp\u003eAuthor contributions\u003c/p\u003e\n\u003cp\u003eKuldeep Kumar Pandit designed the benchmark, formulated the ten cross-domain questions and four-criterion scoring rubric, administered all three runs under blinded conditions, scored all model outputs, conducted the statistical analysis, and drafted the manuscript in its entirety.\u003c/p\u003e\n\u003cp\u003eData availability\u003c/p\u003e\n\u003cp\u003eThe full text of all model responses (180 iterations), the complete scoring rubric with criterion-level justifications, the raw scoring matrix, and all statistical code are reported in the supplementary materials accompanying this submission. No proprietary or non-public data were used. All queries were submitted to publicly available consumer interfaces; no API access or proprietary data pipelines were used.\u003c/p\u003e\n\u003cp\u003eGenerative AI use disclosure\u003c/p\u003e\n\u003cp\u003eThe models evaluated in this study (Claude Sonnet 4.6, ChatGPT-5.2, Gemini 3 Flash, Grok 4.2, DeepSeek R1, and Perplexity powered by Grok 4.1) are themselves the subject of investigation. Their outputs constitute the raw data of the study. No generative AI was used to write, edit, or revise the manuscript text.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n \u003cli\u003eHendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., \u0026amp; Steinhardt, J. (2021). Measuring massive multitask language understanding. arXiv:2009.03300.\u003c/li\u003e\n \u003cli\u003eLiang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., \u0026hellip; \u0026amp; Leskovec, J. (2022). Holistic evaluation of language models. arXiv:2211.09110.\u003c/li\u003e\n \u003cli\u003eSrivastava, A., Rastogi, A., Rao, A., Shoaib, A. O. M., Torralba, A., Hashimoto, T., \u0026hellip; \u0026amp; Wu, T. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615.\u003c/li\u003e\n \u003cli\u003eOstrom, E. (1990). Governing the Commons: The Evolution of Institutions for Collective Action. Cambridge University Press.\u003c/li\u003e\n \u003cli\u003eGigerenzer, G., \u0026amp; Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684\u0026ndash;704.\u003c/li\u003e\n \u003cli\u003eAiello, L. C., \u0026amp; Wheeler, P. (1995). The Expensive-Tissue Hypothesis: The brain and the digestive system in human and primate evolution. Current Anthropology, 36(2), 199\u0026ndash;221.\u003c/li\u003e\n \u003cli\u003ePortugal, S. J., Hubel, T. Y., Fritz, J., Heese, S., Trobe, D., Voelkl, B., \u0026hellip; \u0026amp; Usherwood, J. R. (2014). Upwash exploitation and downwash avoidance by flap phasing in ibis formation flight. Nature, 505, 399\u0026ndash;402.\u003c/li\u003e\n \u003cli\u003eEddy, D. M. (1982). Probabilistic reasoning in clinical medicine: Problems and opportunities. In D. Kahneman, P. Slovic, \u0026amp; A. Tversky (Eds.), Judgment Under Uncertainty: Heuristics and Biases (pp. 249\u0026ndash;267). Cambridge University Press.\u003c/li\u003e\n \u003cli\u003eHardin, G. (1968). The tragedy of the commons. Science, 162(3859), 1243\u0026ndash;1248.\u003c/li\u003e\n \u003cli\u003eKadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., \u0026hellip; \u0026amp; Kaplan, J. (2022). Language models (mostly) know what they know. arXiv:2207.05221.\u003c/li\u003e\n \u003cli\u003eLin, S., Hilton, J., \u0026amp; Evans, O. (2022). TruthfulQA: Measuring how models mimic human falsehoods. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 3214\u0026ndash;3252.\u003c/li\u003e\n \u003cli\u003eGuo, C., Pleiss, G., Sun, Y., \u0026amp; Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning (ICML), 1321\u0026ndash;1330.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"large language model evaluation, self-calibration, epistemic metacognition, benchmark, cross-domain reasoning, self-audit reliability, AI safety","lastPublishedDoi":"10.21203/rs.3.rs-8937831/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8937831/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAccurate self-knowledge — the capacity to know what one does not know — may be as important as raw capability in deployed artificial intelligence systems. We report a pre-registered, three-run, blinded evaluation of six frontier large language models (Claude, ChatGPT, Gemini, Grok, DeepSeek, and Perplexity) on ten cross-domain questions spanning thermodynamics, limnology, biostatistics, cardiovascular physiology, evolutionary biology, atmospheric chemistry, Bayesian statistics, evolutionary anthropology, fluid dynamics, and game theory. Each question required not only a correct conclusion but a complete multi-criterion reasoning chain; partial credit was awarded for incomplete but directionally correct reasoning. The same protocol was administered across three independent runs (n = 180 total iterations). Two outcome measures were collected: (1) independently-scored answer quality (0.0 / 0.5 / 1.0 per question per run) and (2) self-audit accuracy — the model's own verdict on its performance against disclosed ground truth. Models differed substantially in mean answer quality (range 7.17–9.17/10; Kruskal-Wallis H = 8.49, p = 0.13; Cohen's d Claude vs Perplexity = 6.93). Critically, self-audit reliability varied even more dramatically: Claude achieved near-perfect self-calibration (Spearman ρ = 0.90, mean discrepancy +0.33), while Grok exhibited extreme systematic deflation (mean discrepancy −7.5), DeepSeek showed run-to-run incoherence (discrepancies +0.5, −3.5, +2.5 across identical-quality answers), and Perplexity lapsed into hallucination for one domain across two runs. Four of ten questions achieved universal 100% success; four questions produced partial success rates of 72–83%, revealing structurally embedded knowledge gaps resistant to correction by ground-truth exposure alone. Token verbosity showed no positive correlation with accuracy (Pearson r = 0.28, p = 0.15), while Gemini achieved the highest token efficiency (6.16 accuracy points per 1,000 tokens). The results suggest that current LLMs possess stratified epistemic competencies: a reliable core of common-curriculum reasoning that is fully consistent, surrounded by domains where mechanism-level specificity remains systematically incomplete. Self-audit performance is an independent dimension of capability that diverges sharply from answer quality and constitutes a significant dimension for model evaluation.\u003c/p\u003e","manuscriptTitle":"Measuring What AI Models Know and Whether They Know What They Don't Know: A Three-Run, Blind, Cross-Domain Benchmark of Six Leading Large Language Models","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-16 02:41:04","doi":"10.21203/rs.3.rs-8937831/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e18fcae6-84ad-484f-b768-d021474ce726","owner":[],"postedDate":"March 16th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":64425830,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":64425831,"name":"Biological sciences/Evolution"},{"id":64425832,"name":"Physical sciences/Mathematics and computing"},{"id":64425833,"name":"Biological sciences/Psychology"},{"id":64425834,"name":"Social science/Psychology"}],"tags":[],"updatedAt":"2026-03-16T02:41:04+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-16 02:41:04","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8937831","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8937831","identity":"rs-8937831","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}