Systematic Evaluation of Multilingual Retrieval-Augmented Generation for Gastrointestinal Tumor Board Decision Support

Systematic Evaluation of Multilingual Retrieval-Augmented Generation for Gastrointestinal Tumor Board Decision Support | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Systematic Evaluation of Multilingual Retrieval-Augmented Generation for Gastrointestinal Tumor Board Decision Support Derna Stifini, Andrea Della Penna, André L. Mihaljevic, Pavlos Missios, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8849187/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 18 You are reading this latest preprint version Abstract Large language models (LLMs) have been proposed as decision support tools for multidisciplinary tumor boards, yet systematic preclinical validation of retrieval-augmented generation (RAG) pipelines remains lacking. In this retrospective framework validation study using real-world clinical data, we applied a modular evaluation framework to 100 gastrointestinal tumor board cases spanning five cancer types, systematically testing 16 configurations varying model variant, multilingual retrieval strategy, query formulation, and corpus scope. Baseline concordance with multidisciplinary team recommendations ranged from 79–85%. Combining query rewriting with curated guideline retrieval improved concordance to 93–95% (p < 0.01), with prompt design and corpus curation exerting greater influence than model selection. Among residual discordant cases in optimal configurations, approximately 60% represented clinically inappropriate recommendations rather than acceptable therapeutic alternatives. These findings demonstrate that systematic RAG optimization substantially improves clinical decision support concordance, while the high rate of inappropriate residual errors underscores the necessity of mandatory expert oversight before any clinical deployment. Biological sciences/Cancer Biological sciences/Computational biology and bioinformatics Health sciences/Oncology Figures Figure 1 Figure 2 Introduction Multidisciplinary tumor boards (MDT) represent the gold standard for complex cancer treatment decisions, integrating expertise from surgery, medical oncology, radiation oncology, pathology, and radiology. Increasing case volumes and growing clinical complexity place substantial operational pressure on these boards, often leading to delays between case presentation and treatment initiation—an established bottleneck in cancer care delivery ( 1 – 4 ). Large language models (LLMs) such as GPT have been proposed as tools to support clinical decision-making. Prior studies comparing LLM-generated recommendations with MDT decisions across multiple malignancies have reported moderate concordance rates. In gastrointestinal oncology, agreement is generally higher for broad treatment strategies than for specific therapeutic regimens, highlighting the limitations of standalone prompting in complex oncological decision-making ( 5 – 8 ). Retrieval-augmented generation (RAG) has emerged as a promising approach to address these limitations by grounding LLM outputs in external, domain-specific knowledge such as clinical practice guidelines ( 9 – 13 ). By combining language model inference with targeted retrieval, RAG aims to improve accuracy, transparency, and currency of clinical recommendations. However, existing studies typically evaluate single RAG configurations against non-retrieval baselines and provide limited insight into which system components—model selection, retrieval strategy, or prompt engineering—drive performance gains ( 14 , 15 ). Moreover, several key challenges remain insufficiently explored. First, most evaluations are conducted in a single language, despite real-world clinical documentation and guideline ecosystems being inherently multilingual—particularly in non-English-speaking healthcare systems ( 15 , 16 ). Second, few studies distinguish between recommendations that are clinically inappropriate and those representing acceptable therapeutic alternatives ( 14 , 16 ). Third, systematic comparisons between custom RAG pipelines and commercially available solutions, such as ChatGPT's native file retrieval, are lacking ( 16 – 18 ). To address these gaps, we developed a comprehensive evaluation framework for gastrointestinal oncology MDT decision support. Using a factorial design, we systematically tested 16 configurations varying the model variant, retrieval interface, query formulation, and corpus scope across 100 real-world MDT cases. We quantified the relative contribution of each component, analysed how retrieval strategies reshape model inputs, and conducted expert review of discordant recommendations to distinguish acceptable alternatives from clinically inappropriate outputs. Beyond evaluating a specific RAG system, this work establishes a generalizable methodology for rigorous clinical LLM validation. The framework—combining factorial experimental design, retrieval analysis, and expert appropriateness review—provides a template for assessing LLM decision support systems across oncology and other high-stakes clinical domains where safety and clinical appropriateness remain paramount. Results Cohort Characteristics The cohort comprised 100 patients (median age 64.5 years, IQR 56–71; 64% male), equally distributed across five gastrointestinal cancer types: esophageal, gastric, colorectal, hepatobiliary, and pancreatic (n = 20 each). Demographics varied by tumor type: esophageal cancer showed greatest male predominance (75%), while gastric cancer was evenly distributed (50% male). Systemic therapy and surgery were most commonly recommended (29% each), followed by multistep strategies (19%). Treatment patterns varied substantially by tumor type and presentation timing ( Supplementary Table S1 in Supplementary Materials ). Power Analysis Power to detect differences varied substantially across framework comparisons. Comparisons involving retrieval-augmented frameworks with query rewriting showed high power (> 0.90) to detect meaningful differences due to pronounced discordance asymmetry. Closely related configurations exhibited symmetric discordance patterns and low power (~ 0.03). Most other comparisons fell in an intermediate range (~ 0.3–0.6). Detailed power estimates are provided in Supplementary Results ( Supplementary Figure S1 in Supplementary Materials ). Overall Concordance with the MDT Recommendation Concordance between LLM outputs and MDT recommendations varied across configurations (Fig. 1 ). Cochran's Q test confirmed overall differences (Q = 49.296, df = 15, p < 0.001). Baseline configurations without query rewriting and without retrieval achieved concordance rates of 79% for GPT-4o-mini and 85% for GPT-4o. Retrieval-augmented configurations combining query rewriting with selected guideline corpora achieved the highest concordance: 93% for GPT-4o-mini (p = 0.002 vs. baseline) and 95% for GPT-4o (p = 0.001 vs. baseline). In absolute terms, the number of discordant cases was reduced from 21 to 7 (GPT-4o-mini) and from 15 to 5 (GPT-4o) across 100 cases. In contrast, ChatGPT Assistant configurations showed no significant improvement over baseline despite accessing the same source guidelines (GPT-4o-mini: 80%, GPT-4o: 83%), performing significantly worse than custom RAG pipelines (p < 0.01). The combination of query rewriting and curated retrieval was critical for performance. Query rewriting alone achieved 87% concordance (GPT-4o), and retrieval alone achieved 86%, neither differing significantly from the baseline. Only the integrated use of rewriting and curated retrieval consistently resulted in a significant improvement, underscoring the importance of optimizing the full pipeline. Full results for all configurations, including 95% confidence intervals, are provided in Supplementary Materials ( Supplementary Table S2 in Supplementary Materials ). Retrieval Characteristics Jaccard similarity analysis quantified how different retrieval strategies influenced which guideline content the model received (Table 1 ). Compared to baseline configuration (original query, full corpus), using selected corpora alone produced a Jaccard similarity index of 0.31, indicating substantial but incomplete overlap in retrieved content. Query rewriting alone produced a Jaccard index of 0.15, while the combined approach achieved 0.08—minimal overlap demonstrating that combined optimization fundamentally reshapes information retrieval. Table 1 Jaccard similarity indices comparing retrieved guideline chunks across framework configurations. Values represent mean Jaccard similarity computed across all 100 cases. Lower values indicate greater divergence in retrieved content, demonstrating that query reformulation and corpus curation substantially alter information retrieval patterns. Configuration Comparison Jaccard Index Interpretation Full vs. Selected corpora (original queries) 0.31 Moderate overlap Rewritten queries (full vs. selected corpora) 0.24 Low-moderate overlap Original vs. Rewritten queries (full corpora) 0.15 Low overlap Original/Full vs. Rewritten/Selected 0.08 Minimal overlap Notably, rewritten queries retrieved more similar content across corpora types (Jaccard = 0.24) than original versus rewritten queries on the same corpus (0.15). This pattern indicates that query reformulation influences retrieval more than corpora size reduction—a finding with important implications for framework design priorities. The mean number of retrieved graphical elements per case varied markedly by retrieval strategy (Table 2 ). Baseline configuration retrieved 0.39 graphical elements per case on average. Query rewriting alone increased this to 1.04, while curated corpus alone retrieved 1.07. The combined approach achieved 2.56 graphical elements per case, a 6.6-fold increase over baseline. This substantial improvement suggests that combined optimization captures visual clinical algorithms (flowcharts and decision trees) that likely contribute to enhanced recommendation concordance, as these graphical elements often encode complex treatment pathways more effectively than text alone. Table 2 Mean number of retrieved graphical elements per case by retrieval strategy. Configuration Mean Graphical Elements Fold-Change vs. Baseline Baseline (original query, full corpora) 0.39 1.0× Query rewriting only 1.04 2.7× Selected corpora only 1.07 2.7× Combined optimization 2.56 6.6× Evaluation of Discordant Recommendations Among discordant cases in the best-performing configurations, expert review evaluated whether LLM recommendations were clinically acceptable alternatives or medically inappropriate. For GPT-4o-mini (7 discordant cases from 93% concordance), 4/7 (57%) were rated "off-base"—medically inappropriate given the presented information—while 3/7 (43%) represented reasonable alternative treatments. For GPT-4o (5 discordant cases from 95% concordance), 3/5 (60%) were "off-base" and 2/5 (40%) clinically acceptable alternatives. Common failure modes in off-base recommendations included proposing immediate surgery for tumors unresectable or borderline on imaging, initiating systemic therapy without confirmed malignancy or before essential diagnostics, and prioritizing liver transplantation without considering primary local bridging therapies. Other errors involved suggesting curative or surgical approaches despite ongoing systemic response or unresectable metastatic disease. Performance by Tumor Type, Consultation Type, and Recommendation Type Detailed subgroup analyses are provided in Supplementary Results ( Supplementary Figures S2–S5 in Supplementary Materials ). Key findings include substantial variation by cancer type (75–100% concordance), with esophageal and pancreatic cancers achieving near-perfect concordance in optimal configurations while hepatobiliary cancers proved most challenging. Follow-up consultations demonstrated higher baseline concordance than first presentations, though optimized frameworks substantially narrowed this gap. Systemic therapy recommendations achieved consistently high concordance (≥ 90%) across configurations, whereas localized therapies showed greater variability (50–88%). Framework Consistency Analysis To identify which cases proved universally challenging versus those where framework selection determined success, we classified each case's performance as "Mostly Wrong" (≤ 30% frameworks concordant), "Mostly Correct" (> 90%), or "Fully Concordant" (100%). Four cases (4%) were classified as "Mostly Wrong," failing across frameworks regardless of configuration. All involved colorectal (2/4) or hepatobiliary (2/4) cancers with complex clinical scenarios including liver metastases in cirrhotic patients and ambiguous diagnostic findings. Conversely, 48 cases (48%) achieved perfect agreement across all frameworks, and 66 cases (66%) showed > 90% concordance. Discussion We developed a systematic framework for evaluating retrieval-augmented generation (RAG) combined with prompt engineering for gastrointestinal MDT decision support. By decomposing performance across 16 configurations, we isolated the relative impact of model choice, retrieval strategy, and prompt engineering — moving beyond single-configuration comparisons to provide actionable insights for system design. We showed that optimization of query formulation and retrieval strategy had the largest impact on performance. We further demonstrate that, in a multilingual context, query reformulation reshapes model inputs more profoundly than corpus size reduction. Moreover, the inclusion of transcribed graphical elements likely contributed to improved performance by capturing visual clinical algorithms. Finally, expert assessment of discordant recommendations distinguishes clinically inappropriate outputs from acceptable alternatives, offering a safety-oriented perspective that extends beyond concordance metrics alone. Optimization of query formulation and retrieval strategy consistently influenced outcomes more than model selection. Both GPT-4o and GPT-4o-mini showed comparable results across configurations when provided equivalent inputs. The synergistic benefit of combining retrieval augmentation with prompt rewriting represents a key finding: while each approach individually showed trends toward improvement, only their combination consistently achieved statistical significance ( 19 – 21 ). Jaccard similarity analysis quantified how retrieval strategy fundamentally shapes model inputs: minimal overlap (Jaccard index 0.08) when combining query rewriting with curated corpora demonstrates that framework adjustments produce substantial changes in what information the model receives. Query reformulation influenced retrieval patterns more than corpus size reduction (Jaccard 0.15 vs. 0.24), highlighting prompt engineering's critical role in determining retrieval effectiveness ( 10 , 12 , 22 ). A distinctive aspect of our framework is its multilingual deployment. The guideline corpus included documents in both German and English. German clinical documentation was used unchanged in the baseline configuration. In the rewritten-query configurations, inputs were reformulated and translated into English to standardize case structure and align terminology with guideline content, improving retrieval effectiveness. Final recommendations were consistently generated in German. This cross-lingual setup reflects real-world clinical environments but is underexplored in prior RAG evaluations ( 16 , 23 , 24 ). Our findings suggest that query reformulation — including translation — has a stronger impact on retrieval than corpus size alone, though translation and reformulation effects cannot be fully disentangled and warrant further investigation ( 25 , 26 ). Contrary to the notion that more data automatically enhances LLM performance, curated corpora improved outcomes by reducing noise and focusing retrieval on treatment-relevant content while excluding diagnostic algorithms, epidemiological background, and prevention strategies ( 9 , 12 , 27 – 33 ). Despite access to identical guideline sources, ChatGPT Assistant configurations did not outperform baseline approaches, performing significantly worse than custom RAG pipelines. This failure is unlikely to reflect guideline content per se, given that both systems accessed the same documents. While we cannot directly inspect ChatGPT Assistant's internal retrieval mechanism, the observed pattern is consistent with a dilution effect arising from whole-document ingestion: rather than applying targeted retrieval, the assistant processes entire PDFs as undifferentiated context, relying on internal attention mechanisms to surface relevant passages. In contrast, our custom pipeline applied hierarchical chunking, semantic embedding with a multilingual model, and targeted top-k retrieval. When guidelines are long and structurally heterogeneous — as clinical practice guidelines invariably are — this architectural difference may explain why whole-document ingestion dilutes the signal from treatment-relevant passages with epidemiological background, diagnostic algorithms, and prevention content ( 10 , 34 ). This interpretation is consistent with our corpus curation finding: even within our custom RAG, excluding non-treatment sections improved concordance substantially. Unrestricted document access, whether through commercial assistants or uncurated corpora, appears insufficient for reliable clinical decision support without deliberate retrieval optimization. Moreover, the inclusion of transcribed graphical elements likely contributed to improved performance by capturing visual clinical algorithms ( 11 ). This finding indicates that unrestricted document access alone is insufficient for reliable clinical decision support. Systematic retrieval optimization remains necessary, even when commercial solutions provide native file-based reference capabilities. The high rate of clinically inappropriate recommendations among discordant cases (~ 60%) reveals a critical limitation of current LLM systems: inability to integrate nuanced patient-specific factors including performance status, comorbidity burden, and patient treatment preferences. Guidelines provide decision frameworks but cannot encode every clinical nuance. Expert oncologists weigh factors like frailty assessment, social support systems, and goals of care that are difficult to capture in structured clinical documentation. In oncology, inappropriate recommendations may result in treatment delays that allow disease progression, unnecessary toxicity from contraindicated therapies, or missed curative opportunities. Even with 95% concordance, the remaining 5% of cases may include life-threatening errors that could harm patients if accepted without expert review ( 20 , 35 ). This finding has profound implications for deployment—LLM decision support must remain advisory rather than autonomous, with expert oversight as a mandatory safety mechanism. High concordance rates alone do not guarantee safe clinical practice, making appropriateness-based evaluation essential alongside concordance metrics. This evaluation methodology is not limited to gastrointestinal oncology. The framework—combining factorial design, retrieval analysis, and expert appropriateness review—can be adapted to other clinical domains where guideline-based decision support is feasible and safety-critical evaluation is required. Substantial performance variation across tumor types (70–100%) underscores that LLM framework validation must be context-specific rather than assumed to generalize across oncology domains ( 27 , 36 , 37 ). Tumor entities governed by highly standardized treatment pathways, such as esophageal and pancreatic cancers, achieved near-perfect concordance, whereas clinically heterogeneous settings, including hepatobiliary and colorectal malignancies, required more extensive framework optimization to reach comparable performance ( 9 , 38 – 42 ). This sensitivity to clinical heterogeneity extended beyond tumor type to documentation context. Baseline concordance was consistently lower for first presentations than for follow-up cases (76–82% vs. 82–88%), reflecting greater variability and incompleteness in initial documentation. With optimized configurations combining query rewriting and curated RAG, this gap largely disappeared, with concordance increasing to 94% for first presentations and 96% for follow-ups. These findings indicate that appropriate framework design can partially compensate for input variability, a critical consideration for real-world deployment where documentation quality is inherently inconsistent ( 31 , 43 ). At the level of individual treatment recommendations, performance differences further highlight the need for adaptive system behavior. Systemic therapy recommendations achieved high concordance even under baseline conditions (~ 90%), whereas localized treatment recommendations showed substantially lower baseline concordance (50%) ( 28 , 44 ). Rather than applying uniform retrieval strategies across all clinical questions, future systems could dynamically adjust retrieval depth and evidence aggregation based on decision complexity. Such adaptive workflows may improve the accuracy–efficiency tradeoff by reserving deeper, multi-source retrieval for clinically complex scenarios while maintaining lightweight retrieval for straightforward, guideline-aligned recommendations. One promising application involves AI-assisted therapeutic triage to streamline specialist referrals. In current practice, patients enter the system when a clinician (surgeon, oncologist, or other specialist) collects documentation and completes a structured case report. This case is then scheduled for MDT discussion, and only afterward is the patient referred to the appropriate therapeutic specialist. As MDT waiting lists grow longer, this workflow can create substantial delays between initial presentation and treatment initiation ( 1 – 4 ). In this context, our study evaluates a single functional component that could underpin a more comprehensive AI-assisted MDT workflow: guideline-aligned therapeutic categorization. Using prompt engineering and retrieval-augmented generation, we assessed whether this step is technically reliable enough to merit further development, rather than proposing a fully autonomous agent-based system. An AI system could categorize cases into broad therapeutic domains at the initial entry point, enabling immediate specialist referral while maintaining expert oversight. The receiving specialist would retain authority to either proceed with the AI-suggested pathway or escalate complex or uncertain cases to full MDT review, reserving comprehensive multidisciplinary discussion for genuinely ambiguous scenarios. This hybrid workflow could reduce time-to-treatment for straightforward cases while ensuring that challenging cases still benefit from collective expertise. In this setting, the observed failure patterns do not argue against clinical use per se, but rather delineate the conditions under which deployment may be appropriate. The high proportion of clinically inappropriate outputs among discordant cases reinforces that LLM-based systems should function as conservative triage and escalation tools, rather than autonomous decision-makers. Viewed in this light, guideline-aligned therapeutic categorization is best understood as a conservative triage and escalation component within a larger clinical workflow, rather than as a stand-alone or autonomous decision system. The translation of this capability into measurable clinical benefit will require validation of the complete end-to-end process, including specialist acceptance, MDT escalation rates, effects on time-to-treatment, and downstream patient outcomes. Our study has several limitations that point toward important research priorities. First, our single-center retrospective design limits generalizability. The cohort derives from a German university hospital with specific semi-structured documentation practices, patient demographics, and guideline selection (primarily NCCN and German S3 guidelines), which prevents assessment of real-world workflow integration across diverse settings. External validation across multiple institutions, languages, and patient populations is essential to assess reproducibility and identify algorithmic biases. Multi-center prospective trials could establish real-world performance under varying documentation quality and clinical workflows, ultimately measuring patient-centered outcomes including time-to-treatment, decision quality metrics, clinician cognitive load, and patient satisfaction. Second, our multilingual embedding approach introduces potential confounding between translation effects and query reformulation benefits. In the rewritten-query configurations, we converted German clinical documentation to English for guideline retrieval, making it difficult to fully separate language-related factors from query optimization effects. Controlled comparisons should explicitly quantify translation-related performance impacts across different language pairs and clinical contexts. Third, subgroup analyses had limited statistical power. While adequately powered for primary analyses comparing major configuration differences (power > 0.90 for retrieval-augmented frameworks with query rewriting), tumor-type subgroups (n = 20 each) were exploratory with limited power for detecting moderate effects. Larger validation studies are needed to establish cancer-specific performance benchmarks with adequate precision and to develop adaptive frameworks that dynamically adjust retrieval depth based on tumor type, case complexity, and uncertainty estimates. Fourth, concordance was assessed at the level of therapeutic category rather than specific regimen or intent — a deliberate design choice reflecting the nature of retrospective MDT documentation, which was not originally structured for granular binary classification and often lacked the detailed therapeutic information required for it. This categorical approach provides a clinically meaningful and stable benchmark for comparing framework configurations, which was the primary focus of our study. Reviewer blinding to framework configurations and original recommendations further reduced classification bias. Nevertheless, this simplification may introduce both over- and underestimation of true alignment: categorical concordance does not capture intra-category differences in regimen, dosing, or therapeutic intent, and the liberal matching rule (concordant if the LLM matched any MDT-accepted option) may overestimate agreement in cases where the MDT documented multiple alternatives. Expert review of discordant cases partially addresses this by distinguishing clinically inappropriate outputs from acceptable alternatives. Prospective clinical trials measuring patient-centered outcomes — including time-to-treatment, decision quality, and alignment with patient preferences — remain essential to establish real-world clinical validity. Fifth, the use of proprietary models raises reproducibility and governance concerns. GPT-4o and GPT-4o-mini introduce issues related to data privacy, reproducibility challenges due to model updates, and vendor dependence ( 45 ). Investigation of open-source model alternatives that provide greater transparency and institutional control represents an important research direction, though current open models lag commercial systems in reasoning capability ( 46 ). Furthermore, results are specific to the model snapshots used (gpt-4o-2024-11-20 and gpt-4o-mini-2024-07-18); performance may differ as models are updated, and periodic re-evaluation against fixed snapshots is advisable before clinical deployment. All raw LLM outputs are preserved in the public repository to support future comparisons as model capabilities evolve. Sixth, current guideline formats are not optimized for machine retrieval. Guidelines are designed for human readers rather than computational systems. Development of standards for guideline structuring and maintenance could optimize machine retrievability through standardized section labeling, explicit decision trees, and structured tables encoding treatment criteria. Implementation of continuous corpus updating mechanisms would maintain currency as new evidence emerges and guidelines evolve. Finally, system transparency and clinician trust remain underexplored. Our evaluation focused on concordance and appropriateness but did not assess clinician trust or usability in real-world settings. Creation of user-centered interfaces exposing retrieval trails would support clinician oversight and trust. Transparency about which guideline sections informed recommendations enables expert validation and iterative refinement. These findings suggest that advancing clinical AI requires equal attention to workflow engineering, data curation, and human-AI collaboration as to model development itself. The evaluation framework presented here provides a foundation for rigorous assessment of LLM decision support systems across oncology and other high-stakes clinical domains. By demonstrating both the potential of optimized RAG frameworks to improve concordance and the persistent safety challenges requiring mandatory expert validation, this work establishes a path toward responsible clinical AI deployment that prioritizes patient safety alongside technical performance. Methods Study Design and Case Selection This retrospective study was approved by the Institutional Ethics Committee of the University Hospital Tübingen (Protocol No. 273/2024BO1). The cohort included patients discussed at the gastrointestinal MDT of our institution between March 2022 and December 2023. All clinical data were fully anonymized prior to analysis. Patients enrolled in clinical trials were excluded to minimize protocol-driven bias. The study followed STROBE reporting guidelines. Five tumor subgroups were defined according to German S3 guidelines: esophageal, gastric, pancreatic, hepatobiliary, and colorectal cancer. Using simple random sampling stratified by tumor type and presentation timing, 20 patients per subgroup were selected (10 first presentations and 10 follow-up cases), resulting in 100 cases overall. Clinical data were extracted from standardized MDT templates including patient demographics, tumor characteristics, staging information, prior treatments, and current clinical question. Cases with mixed diagnoses not clearly fitting standard guideline pathways, as well as patients referred to or enrolled in clinical trials, were excluded to create a corpus aligned with guideline-based treatment algorithms. Framework Architecture A modular framework was developed to evaluate four components in factorial combination (Fig. 2 ): LLM variant (GPT-4o vs. GPT-4o-mini) Retrieval interface (custom API-based RAG vs. ChatGPT Assistant) Query formulation (original German case notes vs. rewritten structured queries) Guideline corpus scope (full corpus vs. curated treatment-focused subset) This 2×2×2×2 factorial design resulted in 16 distinct configurations ( Supplementary Table S3 in Supplementary Materials ), enabling systematic assessment of how individual components influenced concordance with MDT recommendations. All experiments were conducted using the following production snapshots: gpt-4o-2024-11-20 and gpt-4o-mini-2024-07-18. Query Formulation For each case, LLMs received structured, anonymized MDT reports containing clinical and oncological variables. Inputs consisted exclusively of structured text extracted from MDT documentation. No imaging files, PDFs, or free-text physician correspondence were provided. For rewritten-query configurations, an LLM-in-the-loop approach was used, whereby GPT-4o-mini translated German case descriptions into English and restructured the content to align with oncology guideline terminology and format. This step was implemented to standardize case representation across configurations. Informal review of a representative sample of rewritten queries by a bilingual clinician confirmed that clinical meaning and oncological terminology were preserved, though systematic translation quality assessment was beyond the scope of this study. Representative prompt examples are provided in the Supplementary Materials and in the public repository. Information Retrieval Four retrieval conditions were tested No retrieval (baseline) ChatGPT Assistant with full guideline PDFs uploaded Custom RAG using the complete guideline corpus Custom RAG using a curated corpus All corpora included German S3 guidelines ( Leitlinienprogramm Onkologie ) and NCCN Clinical Practice Guidelines in Oncology relevant to each tumor type, current as of January 2024. The curated corpus excluded sections on epidemiology, prevention, screening, and basic diagnostic workup—content judged by two board-certified oncologists to be irrelevant to treatment planning—to reduce retrieval noise. Custom RAG Implementation Patient cases and guideline documents were embedded using the multilingual BAAI/bge-m3 model. Guideline PDFs were parsed into hierarchical text structures using pymupdf4llm. Tables were converted to JSON format, and graphical elements were transcribed into text descriptions ( 11 ). Hierarchical indexing was implemented using LlamaIndex. For each query, the top five guideline chunks were retrieved based on cosine similarity in embedding space. This approach aimed to balance focused context retrieval with preservation of clinically relevant information. The complete implementation code is available at https://doi.org/10.5281/zenodo.18483659 , with detailed documentation of hyperparameters, preprocessing steps, and retrieval algorithms. Prompting Strategy Prompts followed the C.R.E.A.T.E. framework (Context, Role, Explicit Instruction, Action, Tone, Examples). The model was instructed to act as a MDT and generate a concise German-language treatment recommendation based on the provided case information, with or without retrieved guideline content. For retrieval-augmented configurations, retrieved guideline chunks were appended below the case description. Prompts emphasized actionable treatment recommendations and discouraged deferral or requests for additional diagnostics. Representative prompt templates are provided in the Supplementary Material and in the public repository. Outcome Classification MDT and LLM-generated recommendations were independently classified by two board clinicians, blinded to both the study framework and the original recommendations, into the following nine therapeutic categories: Best supportive care Further diagnostic procedures (without subsequent therapy plan) Endoscopic intervention (e.g., endoscopic resection) Active surveillance or follow-up Multimodal therapy (concurrent use of multiple treatment modalities, such as radiochemotherapy) Multistep therapy (sequential treatment strategies, e.g., neoadjuvant chemotherapy followed by surgery and eventually adjuvant therapy) Surgery (invasive surgical procedures requiring general anesthesia) Systemic therapy (including chemotherapy and immunotherapy) Localized therapy (e.g., radiotherapy, radiofrequency ablation, transarterial chemoembolization, or stereotactic body radiotherapy). Classification was guided by the dominant therapeutic intent. When multiple treatment options were documented by the MDT, LLM recommendations were considered concordant if they matched any accepted option. Disagreements between reviewers were resolved through discussion until consensus was reached. This approach reflects the rate of guideline-consistent recommendations rather than strict one-to-one agreement with MDT decisions. Detailed classification criteria and examples are provided in Supplementary Table S4 in Supplementary Material and in the public repository. Statistical Analysis Concordance rates were calculated for each framework configuration. Overall differences across configurations were assessed using Cochran's Q test, followed by pairwise McNemar tests with continuity correction (α = 0.05). Subgroup analyses examined tumor type, presentation timing, and treatment category. Retrieval similarity between configurations was quantified using the Jaccard similarity index, defined as the intersection divided by the union of retrieved guideline chunks. Power analysis was performed using McNemar's test for paired binary outcomes, assuming asymmetric discordance corresponding to large effect sizes. Sensitivity analyses evaluated power to detect moderate and large differences in concordance with a sample size of 100 cases (see Supplementary Material). Framework-level concordance per case was classified as “Mostly Wrong” (≤ 30%), “Mostly Correct” (> 90%), or “Fully Concordant” (100%), based on predefined thresholds. We quantified the number of retrieved guideline chunks containing graphical elements (e.g. figures and flowcharts) for each retrieval-augmented configuration by automated detection of image and table tags in parsed PDF content. Evaluation of Discordant Recommendations To assess clinical appropriateness of incorrect AI-generated recommendations, we conducted expert evaluation focusing on discordant cases from the best-performing configuration for both model variants. Each recommendation was independently reviewed by two board-certified oncologists who were blinded to model identity. Reviewers classified each recommendation as: "Off-base"—medically inappropriate or not clinically justifiable based on current guidelines and patient characteristics "Alternative"—differing from MDT recommendation but representing a potentially acceptable therapeutic option supported by guidelines or clinical evidence Abbreviations AI Artificial Intelligence HCC Hepatocellular Carcinoma LLM Large Language Model MDT Multidisciplinary Tumor Board RAG Retrieval-augmented generation Declarations Funding: no funding was received. Prior presentation: none. Disclaimer: none. Ethics approval and consent to participate This retrospective study was approved by the Institutional Ethics Committee of the University Hospital Tübingen (Protocol No. 273/2024BO1). The requirement for informed consent was waived by the Ethics Committee due to the retrospective and anonymized nature of the study. Data Availability Patient-level clinical data analysed in this study are subject to institutional data protection policies and cannot be made publicly available. The original patient cases and full MDT recommendations therefore remain restricted clinical datasets. Access to non-anonymized data may be considered on reasonable request, subject to institutional approval, applicable data use agreements, and secure data transfer mechanisms. To support transparency and reproducibility, the authors provide a public repository containing a fully anonymized study dataset and all analysis code. The dataset includes anonymized MDT treatment recommendations, the raw responses generated by the large language models for all 16 experimental frameworks, derived treatment classes, binary concordance labels, and retrieval metadata (retrieved chunk indices and chart counts) enabling replication of concordance and retrieval analyses. Together with the accompanying scripts, these materials allow full methodological reproduction of the study while respecting patient privacy and licensing constraints. The repository is archived with a persistent DOI: https://doi.org/10.5281/zenodo.18483659 Code Availability Complete source code, documentation, and detailed implementation specifications are publicly available in the same repository ( https://doi.org/10.5281/zenodo.18483659 ). The repository includes all preprocessing scripts, embedding and retrieval pipelines, prompt templates, and analysis scripts required to reproduce the experimental setup and the reported results. No proprietary software is required beyond standard Python libraries and publicly documented APIs. Acknowledgments This work received no external funding. The authors acknowledge support from their respective departments and institutions. Author Contributions IC and CE conceived and designed the study. DS and ADP performed data collection and curation. DS, ADP, and IC developed the technical framework and conducted experiments. PM and MB performed expert evaluation of discordant cases. ALM and MB provided clinical oversight and guideline expertise. DS and IC performed statistical analyses. DS and IC drafted the manuscript. All authors critically revised the manuscript and approved the final version. Competing Interests The authors declare no competing financial or non-financial interests. References Braulke F, Kober K, Arndt A, Papendick M, Strauss A, Kramm CM, et al. Optimizing the structure of interdisciplinary tumor boards for effective cancer care. Front Oncol. 2023;13. Chau BL, LaGuardia JS, Kim S, Zhang SC, Pletcher E, Sanford NN, et al. Association of Parental Status and Gender With Burden of Multidisciplinary Tumor Boards Among Oncology Physicians. Jama Network Open. 2023;6(10). Cone EB, Marchese M, Paciotti M, Nguyen DD, Nabi J, Cole AP, et al. Assessment of Time-to-Treatment Initiation and Survival in a Cohort of Patients With Common Cancers. Jama Network Open. 2020;3(12). Oureilidis-DeVivo B. The physician oncology referral process and its effect on tumor board outcomes. J Clin Oncol. 2024;42(16). Erdat EC, Yalçiner M, Örüncü MB, Ürün Y, Senler FÇ. Assessing the accuracy of the GPT-4 model in multidisciplinary tumor board decision prediction. Clin Transl Oncol. 2025. Aghamaliyev U, Karimbayli J, Giessen-Jung C, Matthias I, Unger K, Andrade D, et al. ChatGPT's Gastrointestinal Tumor Board Tango: A limping dance partner? Eur J Cancer. 2024;205. Liao N, Li CK, Gradishar WJ, Klimberg VS, Roshal JA, Yuan TZ, et al. Accuracy and Reproducibility of ChatGPT Responses to Breast Cancer Tumor Board Patients. Jco Clin Cancer Info. 2025;9. Levin G, Gotlieb W, Ramirez P, Meyer R, Brezinov Y. ChatGPT in a gynaecologic oncology multidisciplinary team tumour board: A feasibility study. Bjog-Int J Obstet Gy. 2025;132(1):99–101. Berman E, Sundberg Malek H, Bitzer M, Malek N, Eickhoff C. Retrieval Augmented Therapy Suggestion for Molecular Tumor Boards: Algorithmic Development and Validation Study. J Med Internet Res. 2025;27:e64364. Zhang G, Xu Z, Jin Q, Chen F, Fang Y, Liu Y, et al. Leveraging long context in retrieval augmented language models for medical question answering. NPJ Digit Med. 2025;8(1):239. Kresevic S, Giuffre M, Ajcevic M, Accardo A, Croce LS, Shung DL. Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework. NPJ Digit Med. 2024;7(1):102. Jeong M, Sohn J, Sung M, Kang J. Improving medical reasoning through retrieval and self-reflection with retrieval-augmented large language models. Bioinformatics. 2024;40(Suppl 1):i119–i29. Wang CL, Ong J, Wang CR, Ong H, Cheng RBK, Ong D. Potential for GPT Technology to Optimize Future Clinical Decision-Making Using Retrieval-Augmented Generation. Ann Biomed Eng. 2024;52(5):1115–8. Amugongo LM, Mascheroni P, Brooks S, Doering S, Seidel J. Retrieval augmented generation for large language models in healthcare: A systematic review. PLOS Digit Health. 2025;4(6):e0000877. Neha F, Bhati D, Shukla DK. Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review. Ai-Basel. 2025;6(9). Vach M, Gliem M, Weiss D, Ivan VL, Hauke F, Boschenriedter C, et al. Evaluating Retrieval Augmented Generation-enhanced Large Language Models for Question Answering On German Neurovascular Guidelines. Clin Neuroradiol. 2025. Masanneck L, Meuth SG, Pawlitzki M. Evaluating base and retrieval augmented LLMs with document or online support for evidence based neurology. Npj Digital Medicine. 2025;8(1). Wolk K. Evaluating Retrieval-Augmented Generation Variants for Clinical Decision Support: Hallucination Mitigation and Secure On-Premises Deployment. Electronics-Switz. 2025;14(21). Shah K, Xu AY, Sharma Y, Daher M, McDonald C, Diebo BG, et al. Large Language Model Prompting Techniques for Advancement in Clinical Medicine. J Clin Med. 2024;13(17). Huo B, Boyle A, Marfo N, Tangamornsuksan W, Steen JP, McKechnie T, et al. Large Language Models for Chatbot Health Advice Studies: A Systematic Review. JAMA Netw Open. 2025;8(2):e2457879. Kim TT, Makutonin M, Sirous R, Javan R. Optimizing Large Language Models in Radiology and Mitigating Pitfalls: Prompt Engineering and Fine-tuning. Radiographics. 2025;45(4):e240073. Myers S, Miller TA, Gao Y, Churpek MM, Mayampurath A, Dligach D, et al. Lessons learned on information retrieval in electronic health records: a comparison of embedding models and pooling strategies. J Am Med Inform Assoc. 2025;32(2):357–64. Emssaad I, Ben-Bouazza FE, Tafala I, Chakour El Mezali M, Jioudi B. Leveraging multilingual RAG for breast cancer RCPs: AI-driven speech transcription and compliance in Darija-French clinical discussions. Computer Methods and Programs in Biomedicine Update. 2025;8. Yang R, Ning Y, Keppo E, Liu M, Hong C, Bitterman DS, et al. Retrieval-augmented generation for generative artificial intelligence in health care. npj Health Syst. 2025;2. Névéol A, Dalianis H, Velupillai S, Savova G, Zweigenbaum P. Clinical Natural Language Processing in languages other than English: opportunities and challenges. J Biomed Semant. 2018;9. Gaschi F, Fontaine X, Rastin P, Y. T, editors. Multilingual Clinical NER: Translation or Cross-lingual Transfer? Proceedings of the 5th Clinical Natural Language Processing Workshop; 2023; Toronto, Canada: Association for Computational Linguistics. Benary M, Wang XD, Schmidt M, Soll D, Hilfenhaus G, Nassir M, et al. Leveraging Large Language Models for Decision Support in Personalized Oncology. JAMA Netw Open. 2023;6(11):e2343689. Chen D, Avison K, Alnassar S, Huang RS, Raman S. Medical accuracy of artificial intelligence chatbots in oncology: a scoping review. Oncologist. 2025;30(4). Lammert J, Dreyer T, Mathes S, Kuligin L, Borm KJ, Schatz UA, et al. Expert-Guided Large Language Models for Clinical Decision Support in Precision Oncology. JCO Precis Oncol. 2024;8:e2400478. Lammert J, Dreyer TF, Lörsch AM, Jung J, Lange S, Pfarr N, et al. Large language models for precision oncology: Clinical decision support through expert-guided learning. J Clin Oncol. 2024;42(16). Izacard G, Grave E. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. 16th Conference of the European Chapter of the Association for Computational Linguistics (Eacl 2021). 2021:874–80. Li M, Kilicoglu H, Xu H, Zhang R. BiomedRAG: A retrieval augmented large language model for biomedicine. J Biomed Inform. 2025;162:104769. Poretsky E, Blake VC, Andorf CM, Sen TZ. Assessing the performance of generative artificial intelligence in retrieving information against manually curated genetic and genomic data. Database (Oxford). 2025;2025. Liu S, McCoy AB, Wright A. Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. J Am Med Inform Assoc. 2025;32(4):605–15. Menz BD, Kuderer NM, Bacchi S, Modi ND, Chin-Yee B, Hu T, et al. Current safeguards, risk mitigation, and transparency measures of large language models against the generation of health disinformation: repeated cross sectional analysis. BMJ. 2024;384:e078538. Carl N, Schramm F, Haggenmuller S, Kather JN, Hetz MJ, Wies C, et al. Large language model use in clinical oncology. NPJ Precis Oncol. 2024;8(1):240. Sharma P, Hassan C. Artificial Intelligence and Deep Learning for Upper Gastrointestinal Neoplasia. Gastroenterology. 2022;162(4):1056–66. Chen LC, Zack T, Demirci A, Sushil M, Miao B, Kasap C, et al. Assessing Large Language Models for Oncology Data Inference From Radiology Reports. JCO Clin Cancer Inform. 2024;8:e2400126. Maida M, Celsa C, Lau LHS, Ligresti D, Baraldo S, Ramai D, et al. The Application of Large Language Models in Gastroenterology: A Review of the Literature. Cancers (Basel). 2024;16(19). Shah MA, Kennedy EB, Catenacci DV, Deighton DC, Goodman KA, Malhotra NK, et al. Treatment of Locally Advanced Esophageal Carcinoma: ASCO Guideline. J Clin Oncol. 2020;38(23):2677–94. Liang S, Zhang J, Liu X, Huang Y, Shao J, Liu X, et al. The potential of large language models to advance precision oncology. EBioMedicine. 2025;115:105695. Shool S, Adimi S, Saboori Amleshi R, Bitaraf E, Golpira R, Tara M. A systematic review of large language model (LLM) evaluations in clinical medicine. BMC Med Inform Decis Mak. 2025;25(1):117. Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Adv Neur In. 2020;33. Kaiser KN, Hughes AJ, Yang AD, Turk AA, Mohanty S, Gonzalez AA, et al. Accuracy and consistency of publicly available Large Language Models as clinical decision support tools for the management of colon cancer. J Surg Oncol. 2024;130(5):1104–10. Capobianco I, Della Penna A, Mihaljevic AL, Bitzer M, Eickhoff C, Stifini D. Clinical Accuracy and Safety Concerns Following GPT-5 Public Demonstration in Cancer Care. J Med Syst. 2025;49(1). Can E, Uller W, Kotter E, Vogt K, Doppler M, Bronnimann M, et al. Comparative Evaluation of Proprietary and Open-Source Large Language Models for Systematic Multi-source Information Extraction in Interventional Oncology. Cardiovasc Intervent Radiol. 2025. Additional Declarations No competing interests reported. Supplementary Files ChatGPTVSTumorboardnpjDMsupplementary.docx Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 08 Apr, 2026 Reviews received at journal 07 Apr, 2026 Reviews received at journal 07 Apr, 2026 Reviews received at journal 03 Apr, 2026 Reviews received at journal 01 Apr, 2026 Reviews received at journal 01 Apr, 2026 Reviews received at journal 29 Mar, 2026 Reviewers agreed at journal 10 Mar, 2026 Reviewers agreed at journal 09 Mar, 2026 Reviewers agreed at journal 07 Mar, 2026 Reviewers agreed at journal 04 Mar, 2026 Reviewers agreed at journal 03 Mar, 2026 Reviewers agreed at journal 03 Mar, 2026 Reviewers agreed at journal 02 Mar, 2026 Reviewers invited by journal 02 Mar, 2026 Editor assigned by journal 13 Feb, 2026 Submission checks completed at journal 13 Feb, 2026 First submitted to journal 11 Feb, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8849187","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":600883015,"identity":"5c3ffde6-2afe-44dd-84a6-517af07dd073","order_by":0,"name":"Derna Stifini","email":"","orcid":"","institution":"Universitätsklinikum Tübingen","correspondingAuthor":false,"prefix":"","firstName":"Derna","middleName":"","lastName":"Stifini","suffix":""},{"id":600883016,"identity":"5e00e6bb-7d5c-43a1-93cc-9e304489b478","order_by":1,"name":"Andrea Della Penna","email":"","orcid":"","institution":"Universitätsklinikum Tübingen","correspondingAuthor":false,"prefix":"","firstName":"Andrea","middleName":"Della","lastName":"Penna","suffix":""},{"id":600883017,"identity":"4b42203b-a262-43fd-b5d3-52fa91fad9e1","order_by":2,"name":"André L. Mihaljevic","email":"","orcid":"","institution":"Universitätsklinikum Tübingen","correspondingAuthor":false,"prefix":"","firstName":"André","middleName":"L.","lastName":"Mihaljevic","suffix":""},{"id":600883020,"identity":"688ef92e-06b3-496e-9729-e9e674c62127","order_by":3,"name":"Pavlos Missios","email":"","orcid":"","institution":"Universitätsklinikum Tübingen","correspondingAuthor":false,"prefix":"","firstName":"Pavlos","middleName":"","lastName":"Missios","suffix":""},{"id":600883021,"identity":"ad30d069-7825-4000-9463-e494df8900c5","order_by":4,"name":"Michael Bitzer","email":"","orcid":"","institution":"Universitätsklinikum Tübingen","correspondingAuthor":false,"prefix":"","firstName":"Michael","middleName":"","lastName":"Bitzer","suffix":""},{"id":600883022,"identity":"1e61396d-4b33-4ccd-aecb-d4b7d285bd97","order_by":5,"name":"Carsten Eickhoff","email":"","orcid":"","institution":"Universitätsklinikum Tübingen","correspondingAuthor":false,"prefix":"","firstName":"Carsten","middleName":"","lastName":"Eickhoff","suffix":""},{"id":600883024,"identity":"8c2af135-832b-46ca-88e3-a4aa06d40920","order_by":6,"name":"Ivan Capobianco","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABZElEQVRIie2RwUrDMBjHI4X2kuo1MtheIWPQKg59lYZCd9l0MPBi3SqF7LIH8KCPIHSXsmNHIb30AQY72CH0pDAQhuLFxLkZZfMs2B+UpP366/9LPwAKCv4g6nqHdjyxlPmFxe2yai2L+DeltlbgFuULtFyIt3oBSh+UlV1k5xlwj0/3Sv7Vc9utN+76fqDoowNwgmyWzUaXZRMow0xqDDkmBszu7N+M/dI1c1phytqKnvLGkNPAJE1qh57akWJUGBn8FAoJJsQr6V7cCidNrOgU9QaoaSBCGQkiaCBZSRZc6QnFf+NKw7h//FB4ytlCKD2umC+Sog1ESiwUKlIsYwJXSlPlimthngK+KefIYok4Cz2CzKmGqdMe3wolzU1MaFQNYrUjNVbxtRDN3Qv+x/oPU+jWK0YSD7Mn2gVa385nr7RbwYk/nP8YiQVWI/skkvYxAMrGQVobnwq6WysFBQUF/4V3wZ56sqKLmDcAAAAASUVORK5CYII=","orcid":"","institution":"Universitätsklinikum Tübingen","correspondingAuthor":true,"prefix":"","firstName":"Ivan","middleName":"","lastName":"Capobianco","suffix":""}],"badges":[],"createdAt":"2026-02-11 08:39:34","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8849187/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8849187/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":104182134,"identity":"14684bbf-86e5-4ab5-aed4-6147a1363bd8","added_by":"auto","created_at":"2026-03-08 17:34:07","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":91502,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eOverall concordance rate across 16 framework configurations. Each concordant or non-concordant answer accounts for 1 percentage point. RAG: retrieval-augmented generation, rw: rewritten input. Yellow bars: GPT-4o-mini, blue bars: GPT-4o.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8849187/v1/c11e8fd88c16b40132121174.png"},{"id":104182135,"identity":"5266a7f9-680a-4b3d-b3a5-19cf823fd02c","added_by":"auto","created_at":"2026-03-08 17:34:07","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":77946,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eModular framework for retrieval-augmented generation in oncological decision support, illustrating how query formulation, retrieval strategy, and framework configuration influence concordance between large language model outputs and multidisciplinary tumor board recommendations.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8849187/v1/4f0f257f5876a951bba9a1d6.png"},{"id":104182136,"identity":"82df17b6-ec6c-48ff-b4db-60cea4a3a37c","added_by":"auto","created_at":"2026-03-08 17:34:07","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":1559045,"visible":true,"origin":"","legend":"","description":"","filename":"ChatGPTVSTumorboardnpjDMsupplementary.docx","url":"https://assets-eu.researchsquare.com/files/rs-8849187/v1/fd4f508a2037c5dcd4362db7.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Systematic Evaluation of Multilingual Retrieval-Augmented Generation for Gastrointestinal Tumor Board Decision Support","fulltext":[{"header":"Introduction","content":"\u003cp\u003eMultidisciplinary tumor boards (MDT) represent the gold standard for complex cancer treatment decisions, integrating expertise from surgery, medical oncology, radiation oncology, pathology, and radiology. Increasing case volumes and growing clinical complexity place substantial operational pressure on these boards, often leading to delays between case presentation and treatment initiation\u0026mdash;an established bottleneck in cancer care delivery (\u003cspan additionalcitationids=\"CR2 CR3\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eLarge language models (LLMs) such as GPT have been proposed as tools to support clinical decision-making. Prior studies comparing LLM-generated recommendations with MDT decisions across multiple malignancies have reported moderate concordance rates. In gastrointestinal oncology, agreement is generally higher for broad treatment strategies than for specific therapeutic regimens, highlighting the limitations of standalone prompting in complex oncological decision-making (\u003cspan additionalcitationids=\"CR6 CR7\" citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eRetrieval-augmented generation (RAG) has emerged as a promising approach to address these limitations by grounding LLM outputs in external, domain-specific knowledge such as clinical practice guidelines (\u003cspan additionalcitationids=\"CR10 CR11 CR12\" citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e). By combining language model inference with targeted retrieval, RAG aims to improve accuracy, transparency, and currency of clinical recommendations. However, existing studies typically evaluate single RAG configurations against non-retrieval baselines and provide limited insight into which system components\u0026mdash;model selection, retrieval strategy, or prompt engineering\u0026mdash;drive performance gains (\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eMoreover, several key challenges remain insufficiently explored. First, most evaluations are conducted in a single language, despite real-world clinical documentation and guideline ecosystems being inherently multilingual\u0026mdash;particularly in non-English-speaking healthcare systems (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e). Second, few studies distinguish between recommendations that are clinically inappropriate and those representing acceptable therapeutic alternatives (\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e). Third, systematic comparisons between custom RAG pipelines and commercially available solutions, such as ChatGPT's native file retrieval, are lacking (\u003cspan additionalcitationids=\"CR17\" citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eTo address these gaps, we developed a comprehensive evaluation framework for gastrointestinal oncology MDT decision support. Using a factorial design, we systematically tested 16 configurations varying the model variant, retrieval interface, query formulation, and corpus scope across 100 real-world MDT cases. We quantified the relative contribution of each component, analysed how retrieval strategies reshape model inputs, and conducted expert review of discordant recommendations to distinguish acceptable alternatives from clinically inappropriate outputs.\u003c/p\u003e \u003cp\u003eBeyond evaluating a specific RAG system, this work establishes a generalizable methodology for rigorous clinical LLM validation. The framework\u0026mdash;combining factorial experimental design, retrieval analysis, and expert appropriateness review\u0026mdash;provides a template for assessing LLM decision support systems across oncology and other high-stakes clinical domains where safety and clinical appropriateness remain paramount.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eCohort Characteristics\u003c/p\u003e \u003cp\u003eThe cohort comprised 100 patients (median age 64.5 years, IQR 56\u0026ndash;71; 64% male), equally distributed across five gastrointestinal cancer types: esophageal, gastric, colorectal, hepatobiliary, and pancreatic (n\u0026thinsp;=\u0026thinsp;20 each). Demographics varied by tumor type: esophageal cancer showed greatest male predominance (75%), while gastric cancer was evenly distributed (50% male). Systemic therapy and surgery were most commonly recommended (29% each), followed by multistep strategies (19%). Treatment patterns varied substantially by tumor type and presentation timing (\u003cem\u003eSupplementary Table \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e in Supplementary Materials\u003c/em\u003e).\u003c/p\u003e \u003cp\u003ePower Analysis\u003c/p\u003e \u003cp\u003ePower to detect differences varied substantially across framework comparisons. Comparisons involving retrieval-augmented frameworks with query rewriting showed high power (\u0026gt;\u0026thinsp;0.90) to detect meaningful differences due to pronounced discordance asymmetry. Closely related configurations exhibited symmetric discordance patterns and low power (~\u0026thinsp;0.03). Most other comparisons fell in an intermediate range (~\u0026thinsp;0.3\u0026ndash;0.6). Detailed power estimates are provided in Supplementary Results (\u003cem\u003eSupplementary Figure \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e in Supplementary Materials\u003c/em\u003e).\u003c/p\u003e \u003cp\u003eOverall Concordance with the MDT Recommendation\u003c/p\u003e \u003cp\u003eConcordance between LLM outputs and MDT recommendations varied across configurations (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Cochran's Q test confirmed overall differences (Q\u0026thinsp;=\u0026thinsp;49.296, df\u0026thinsp;=\u0026thinsp;15, p\u0026thinsp;\u0026lt;\u0026thinsp;0.001). Baseline configurations without query rewriting and without retrieval achieved concordance rates of 79% for GPT-4o-mini and 85% for GPT-4o.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e Retrieval-augmented configurations combining query rewriting with selected guideline corpora achieved the highest concordance: 93% for GPT-4o-mini (p\u0026thinsp;=\u0026thinsp;0.002 vs. baseline) and 95% for GPT-4o (p\u0026thinsp;=\u0026thinsp;0.001 vs. baseline). In absolute terms, the number of discordant cases was reduced from 21 to 7 (GPT-4o-mini) and from 15 to 5 (GPT-4o) across 100 cases.\u003c/p\u003e \u003cp\u003eIn contrast, ChatGPT Assistant configurations showed no significant improvement over baseline despite accessing the same source guidelines (GPT-4o-mini: 80%, GPT-4o: 83%), performing significantly worse than custom RAG pipelines (p\u0026thinsp;\u0026lt;\u0026thinsp;0.01).\u003c/p\u003e \u003cp\u003eThe combination of query rewriting and curated retrieval was critical for performance. Query rewriting alone achieved 87% concordance (GPT-4o), and retrieval alone achieved 86%, neither differing significantly from the baseline. Only the integrated use of rewriting and curated retrieval consistently resulted in a significant improvement, underscoring the importance of optimizing the full pipeline.\u003c/p\u003e \u003cp\u003eFull results for all configurations, including 95% confidence intervals, are provided in Supplementary Materials (\u003cem\u003eSupplementary Table S2 in Supplementary Materials\u003c/em\u003e).\u003c/p\u003e \u003cp\u003eRetrieval Characteristics\u003c/p\u003e \u003cp\u003eJaccard similarity analysis quantified how different retrieval strategies influenced which guideline content the model received (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Compared to baseline configuration (original query, full corpus), using selected corpora alone produced a Jaccard similarity index of 0.31, indicating substantial but incomplete overlap in retrieved content. Query rewriting alone produced a Jaccard index of 0.15, while the combined approach achieved 0.08\u0026mdash;minimal overlap demonstrating that combined optimization fundamentally reshapes information retrieval.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cem\u003eJaccard similarity indices comparing retrieved guideline chunks across framework configurations. Values represent mean Jaccard similarity computed across all 100 cases. Lower values indicate greater divergence in retrieved content, demonstrating that query reformulation and corpus curation substantially alter information retrieval patterns.\u003c/em\u003e\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eConfiguration Comparison\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eJaccard Index\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eInterpretation\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFull vs. Selected corpora (original queries)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.31\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eModerate overlap\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRewritten queries (full vs. selected corpora)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.24\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLow-moderate overlap\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOriginal vs. Rewritten queries (full corpora)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLow overlap\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOriginal/Full vs. Rewritten/Selected\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.08\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMinimal overlap\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eNotably, rewritten queries retrieved more similar content across corpora types (Jaccard\u0026thinsp;=\u0026thinsp;0.24) than original versus rewritten queries on the same corpus (0.15). This pattern indicates that query reformulation influences retrieval more than corpora size reduction\u0026mdash;a finding with important implications for framework design priorities.\u003c/p\u003e \u003cp\u003eThe mean number of retrieved graphical elements per case varied markedly by retrieval strategy (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). Baseline configuration retrieved 0.39 graphical elements per case on average. Query rewriting alone increased this to 1.04, while curated corpus alone retrieved 1.07. The combined approach achieved 2.56 graphical elements per case, a 6.6-fold increase over baseline. This substantial improvement suggests that combined optimization captures visual clinical algorithms (flowcharts and decision trees) that likely contribute to enhanced recommendation concordance, as these graphical elements often encode complex treatment pathways more effectively than text alone.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cem\u003eMean number of retrieved graphical elements per case by retrieval strategy.\u003c/em\u003e\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026times;\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eConfiguration\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMean Graphical Elements\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFold-Change vs. Baseline\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBaseline (original query, full corpora)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.39\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026times;\" colname=\"c3\"\u003e \u003cp\u003e1.0\u0026times;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eQuery rewriting only\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1.04\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026times;\" colname=\"c3\"\u003e \u003cp\u003e2.7\u0026times;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSelected corpora only\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1.07\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026times;\" colname=\"c3\"\u003e \u003cp\u003e2.7\u0026times;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCombined optimization\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2.56\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026times;\" colname=\"c3\"\u003e \u003cp\u003e6.6\u0026times;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eEvaluation of Discordant Recommendations\u003c/p\u003e \u003cp\u003eAmong discordant cases in the best-performing configurations, expert review evaluated whether LLM recommendations were clinically acceptable alternatives or medically inappropriate. For GPT-4o-mini (7 discordant cases from 93% concordance), 4/7 (57%) were rated \"off-base\"\u0026mdash;medically inappropriate given the presented information\u0026mdash;while 3/7 (43%) represented reasonable alternative treatments. For GPT-4o (5 discordant cases from 95% concordance), 3/5 (60%) were \"off-base\" and 2/5 (40%) clinically acceptable alternatives.\u003c/p\u003e \u003cp\u003eCommon failure modes in off-base recommendations included proposing immediate surgery for tumors unresectable or borderline on imaging, initiating systemic therapy without confirmed malignancy or before essential diagnostics, and prioritizing liver transplantation without considering primary local bridging therapies. Other errors involved suggesting curative or surgical approaches despite ongoing systemic response or unresectable metastatic disease.\u003c/p\u003e \u003cp\u003ePerformance by Tumor Type, Consultation Type, and Recommendation Type\u003c/p\u003e \u003cp\u003eDetailed subgroup analyses are provided in Supplementary Results (\u003cem\u003eSupplementary Figures S2\u0026ndash;S5 in Supplementary Materials\u003c/em\u003e). Key findings include substantial variation by cancer type (75\u0026ndash;100% concordance), with esophageal and pancreatic cancers achieving near-perfect concordance in optimal configurations while hepatobiliary cancers proved most challenging. Follow-up consultations demonstrated higher baseline concordance than first presentations, though optimized frameworks substantially narrowed this gap. Systemic therapy recommendations achieved consistently high concordance (\u0026ge;\u0026thinsp;90%) across configurations, whereas localized therapies showed greater variability (50\u0026ndash;88%).\u003c/p\u003e \u003cp\u003eFramework Consistency Analysis\u003c/p\u003e \u003cp\u003eTo identify which cases proved universally challenging versus those where framework selection determined success, we classified each case's performance as \"Mostly Wrong\" (\u0026le;\u0026thinsp;30% frameworks concordant), \"Mostly Correct\" (\u0026gt;\u0026thinsp;90%), or \"Fully Concordant\" (100%).\u003c/p\u003e \u003cp\u003eFour cases (4%) were classified as \"Mostly Wrong,\" failing across frameworks regardless of configuration. All involved colorectal (2/4) or hepatobiliary (2/4) cancers with complex clinical scenarios including liver metastases in cirrhotic patients and ambiguous diagnostic findings. Conversely, 48 cases (48%) achieved perfect agreement across all frameworks, and 66 cases (66%) showed\u0026thinsp;\u0026gt;\u0026thinsp;90% concordance.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eWe developed a systematic framework for evaluating retrieval-augmented generation (RAG) combined with prompt engineering for gastrointestinal MDT decision support. By decomposing performance across 16 configurations, we isolated the relative impact of model choice, retrieval strategy, and prompt engineering — moving beyond single-configuration comparisons to provide actionable insights for system design. We showed that optimization of query formulation and retrieval strategy had the largest impact on performance. We further demonstrate that, in a multilingual context, query reformulation reshapes model inputs more profoundly than corpus size reduction. Moreover, the inclusion of transcribed graphical elements likely contributed to improved performance by capturing visual clinical algorithms. Finally, expert assessment of discordant recommendations distinguishes clinically inappropriate outputs from acceptable alternatives, offering a safety-oriented perspective that extends beyond concordance metrics alone.\u003c/p\u003e \u003cp\u003eOptimization of query formulation and retrieval strategy consistently influenced outcomes more than model selection. Both GPT-4o and GPT-4o-mini showed comparable results across configurations when provided equivalent inputs. The synergistic benefit of combining retrieval augmentation with prompt rewriting represents a key finding: while each approach individually showed trends toward improvement, only their combination consistently achieved statistical significance (\u003cspan class=\"CitationRef\"\u003e19\u003c/span\u003e–\u003cspan class=\"CitationRef\"\u003e21\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eJaccard similarity analysis quantified how retrieval strategy fundamentally shapes model inputs: minimal overlap (Jaccard index 0.08) when combining query rewriting with curated corpora demonstrates that framework adjustments produce substantial changes in what information the model receives. Query reformulation influenced retrieval patterns more than corpus size reduction (Jaccard 0.15 vs. 0.24), highlighting prompt engineering's critical role in determining retrieval effectiveness (\u003cspan class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e12\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e22\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eA distinctive aspect of our framework is its multilingual deployment. The guideline corpus included documents in both German and English. German clinical documentation was used unchanged in the baseline configuration. In the rewritten-query configurations, inputs were reformulated and translated into English to standardize case structure and align terminology with guideline content, improving retrieval effectiveness. Final recommendations were consistently generated in German.\u003c/p\u003e \u003cp\u003eThis cross-lingual setup reflects real-world clinical environments but is underexplored in prior RAG evaluations (\u003cspan class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e23\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e24\u003c/span\u003e). Our findings suggest that query reformulation — including translation — has a stronger impact on retrieval than corpus size alone, though translation and reformulation effects cannot be fully disentangled and warrant further investigation (\u003cspan class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e26\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eContrary to the notion that more data automatically enhances LLM performance, curated corpora improved outcomes by reducing noise and focusing retrieval on treatment-relevant content while excluding diagnostic algorithms, epidemiological background, and prevention strategies (\u003cspan class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e12\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e27\u003c/span\u003e–\u003cspan class=\"CitationRef\"\u003e33\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e Despite access to identical guideline sources, ChatGPT Assistant configurations did not outperform baseline approaches, performing significantly worse than custom RAG pipelines. This failure is unlikely to reflect guideline content per se, given that both systems accessed the same documents. While we cannot directly inspect ChatGPT Assistant's internal retrieval mechanism, the observed pattern is consistent with a dilution effect arising from whole-document ingestion: rather than applying targeted retrieval, the assistant processes entire PDFs as undifferentiated context, relying on internal attention mechanisms to surface relevant passages. In contrast, our custom pipeline applied hierarchical chunking, semantic embedding with a multilingual model, and targeted top-k retrieval. When guidelines are long and structurally heterogeneous — as clinical practice guidelines invariably are — this architectural difference may explain why whole-document ingestion dilutes the signal from treatment-relevant passages with epidemiological background, diagnostic algorithms, and prevention content (\u003cspan class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e34\u003c/span\u003e). This interpretation is consistent with our corpus curation finding: even within our custom RAG, excluding non-treatment sections improved concordance substantially. Unrestricted document access, whether through commercial assistants or uncurated corpora, appears insufficient for reliable clinical decision support without deliberate retrieval optimization. Moreover, the inclusion of transcribed graphical elements likely contributed to improved performance by capturing visual clinical algorithms (\u003cspan class=\"CitationRef\"\u003e11\u003c/span\u003e). This finding indicates that unrestricted document access alone is insufficient for reliable clinical decision support. Systematic retrieval optimization remains necessary, even when commercial solutions provide native file-based reference capabilities.\u003c/p\u003e \u003cp\u003eThe high rate of clinically inappropriate recommendations among discordant cases (~ 60%) reveals a critical limitation of current LLM systems: inability to integrate nuanced patient-specific factors including performance status, comorbidity burden, and patient treatment preferences. Guidelines provide decision frameworks but cannot encode every clinical nuance.\u003c/p\u003e \u003cp\u003eExpert oncologists weigh factors like frailty assessment, social support systems, and goals of care that are difficult to capture in structured clinical documentation. In oncology, inappropriate recommendations may result in treatment delays that allow disease progression, unnecessary toxicity from contraindicated therapies, or missed curative opportunities. Even with 95% concordance, the remaining 5% of cases may include life-threatening errors that could harm patients if accepted without expert review (\u003cspan class=\"CitationRef\"\u003e20\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e35\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThis finding has profound implications for deployment—LLM decision support must remain advisory rather than autonomous, with expert oversight as a mandatory safety mechanism. High concordance rates alone do not guarantee safe clinical practice, making appropriateness-based evaluation essential alongside concordance metrics.\u003c/p\u003e \u003cp\u003eThis evaluation methodology is not limited to gastrointestinal oncology. The framework—combining factorial design, retrieval analysis, and expert appropriateness review—can be adapted to other clinical domains where guideline-based decision support is feasible and safety-critical evaluation is required.\u003c/p\u003e \u003cp\u003eSubstantial performance variation across tumor types (70–100%) underscores that LLM framework validation must be context-specific rather than assumed to generalize across oncology domains (\u003cspan class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e36\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e37\u003c/span\u003e). Tumor entities governed by highly standardized treatment pathways, such as esophageal and pancreatic cancers, achieved near-perfect concordance, whereas clinically heterogeneous settings, including hepatobiliary and colorectal malignancies, required more extensive framework optimization to reach comparable performance (\u003cspan class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e38\u003c/span\u003e–\u003cspan class=\"CitationRef\"\u003e42\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThis sensitivity to clinical heterogeneity extended beyond tumor type to documentation context. Baseline concordance was consistently lower for first presentations than for follow-up cases (76–82% vs. 82–88%), reflecting greater variability and incompleteness in initial documentation. With optimized configurations combining query rewriting and curated RAG, this gap largely disappeared, with concordance increasing to 94% for first presentations and 96% for follow-ups. These findings indicate that appropriate framework design can partially compensate for input variability, a critical consideration for real-world deployment where documentation quality is inherently inconsistent (\u003cspan class=\"CitationRef\"\u003e31\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e43\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eAt the level of individual treatment recommendations, performance differences further highlight the need for adaptive system behavior. Systemic therapy recommendations achieved high concordance even under baseline conditions (~ 90%), whereas localized treatment recommendations showed substantially lower baseline concordance (50%) (\u003cspan class=\"CitationRef\"\u003e28\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e44\u003c/span\u003e). Rather than applying uniform retrieval strategies across all clinical questions, future systems could dynamically adjust retrieval depth and evidence aggregation based on decision complexity. Such adaptive workflows may improve the accuracy–efficiency tradeoff by reserving deeper, multi-source retrieval for clinically complex scenarios while maintaining lightweight retrieval for straightforward, guideline-aligned recommendations.\u003c/p\u003e \u003cp\u003eOne promising application involves AI-assisted therapeutic triage to streamline specialist referrals. In current practice, patients enter the system when a clinician (surgeon, oncologist, or other specialist) collects documentation and completes a structured case report. This case is then scheduled for MDT discussion, and only afterward is the patient referred to the appropriate therapeutic specialist. As MDT waiting lists grow longer, this workflow can create substantial delays between initial presentation and treatment initiation (\u003cspan class=\"CitationRef\"\u003e1\u003c/span\u003e–\u003cspan class=\"CitationRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e In this context, our study evaluates a single functional component that could underpin a more comprehensive AI-assisted MDT workflow: guideline-aligned therapeutic categorization. Using prompt engineering and retrieval-augmented generation, we assessed whether this step is technically reliable enough to merit further development, rather than proposing a fully autonomous agent-based system.\u003c/p\u003e \u003cp\u003eAn AI system could categorize cases into broad therapeutic domains at the initial entry point, enabling immediate specialist referral while maintaining expert oversight. The receiving specialist would retain authority to either proceed with the AI-suggested pathway or escalate complex or uncertain cases to full MDT review, reserving comprehensive multidisciplinary discussion for genuinely ambiguous scenarios. This hybrid workflow could reduce time-to-treatment for straightforward cases while ensuring that challenging cases still benefit from collective expertise.\u003c/p\u003e \u003cp\u003eIn this setting, the observed failure patterns do not argue against clinical use per se, but rather delineate the conditions under which deployment may be appropriate. The high proportion of clinically inappropriate outputs among discordant cases reinforces that LLM-based systems should function as conservative triage and escalation tools, rather than autonomous decision-makers.\u003c/p\u003e \u003cp\u003e Viewed in this light, guideline-aligned therapeutic categorization is best understood as a conservative triage and escalation component within a larger clinical workflow, rather than as a stand-alone or autonomous decision system.\u003c/p\u003e \u003cp\u003eThe translation of this capability into measurable clinical benefit will require validation of the complete end-to-end process, including specialist acceptance, MDT escalation rates, effects on time-to-treatment, and downstream patient outcomes.\u003c/p\u003e \u003cp\u003eOur study has several limitations that point toward important research priorities.\u003c/p\u003e \u003cp\u003eFirst, our single-center retrospective design limits generalizability. The cohort derives from a German university hospital with specific semi-structured documentation practices, patient demographics, and guideline selection (primarily NCCN and German S3 guidelines), which prevents assessment of real-world workflow integration across diverse settings. External validation across multiple institutions, languages, and patient populations is essential to assess reproducibility and identify algorithmic biases. Multi-center prospective trials could establish real-world performance under varying documentation quality and clinical workflows, ultimately measuring patient-centered outcomes including time-to-treatment, decision quality metrics, clinician cognitive load, and patient satisfaction.\u003c/p\u003e \u003cp\u003eSecond, our multilingual embedding approach introduces potential confounding between translation effects and query reformulation benefits. In the rewritten-query configurations, we converted German clinical documentation to English for guideline retrieval, making it difficult to fully separate language-related factors from query optimization effects. Controlled comparisons should explicitly quantify translation-related performance impacts across different language pairs and clinical contexts.\u003c/p\u003e \u003cp\u003eThird, subgroup analyses had limited statistical power. While adequately powered for primary analyses comparing major configuration differences (power \u0026gt; 0.90 for retrieval-augmented frameworks with query rewriting), tumor-type subgroups (n = 20 each) were exploratory with limited power for detecting moderate effects. Larger validation studies are needed to establish cancer-specific performance benchmarks with adequate precision and to develop adaptive frameworks that dynamically adjust retrieval depth based on tumor type, case complexity, and uncertainty estimates.\u003c/p\u003e \u003cp\u003eFourth, concordance was assessed at the level of therapeutic category rather than specific regimen or intent — a deliberate design choice reflecting the nature of retrospective MDT documentation, which was not originally structured for granular binary classification and often lacked the detailed therapeutic information required for it. This categorical approach provides a clinically meaningful and stable benchmark for comparing framework configurations, which was the primary focus of our study. Reviewer blinding to framework configurations and original recommendations further reduced classification bias. Nevertheless, this simplification may introduce both over- and underestimation of true alignment: categorical concordance does not capture intra-category differences in regimen, dosing, or therapeutic intent, and the liberal matching rule (concordant if the LLM matched any MDT-accepted option) may overestimate agreement in cases where the MDT documented multiple alternatives. Expert review of discordant cases partially addresses this by distinguishing clinically inappropriate outputs from acceptable alternatives. Prospective clinical trials measuring patient-centered outcomes — including time-to-treatment, decision quality, and alignment with patient preferences — remain essential to establish real-world clinical validity.\u003c/p\u003e \u003cp\u003eFifth, the use of proprietary models raises reproducibility and governance concerns. GPT-4o and GPT-4o-mini introduce issues related to data privacy, reproducibility challenges due to model updates, and vendor dependence (\u003cspan class=\"CitationRef\"\u003e45\u003c/span\u003e). Investigation of open-source model alternatives that provide greater transparency and institutional control represents an important research direction, though current open models lag commercial systems in reasoning capability (\u003cspan class=\"CitationRef\"\u003e46\u003c/span\u003e). Furthermore, results are specific to the model snapshots used (gpt-4o-2024-11-20 and gpt-4o-mini-2024-07-18); performance may differ as models are updated, and periodic re-evaluation against fixed snapshots is advisable before clinical deployment. All raw LLM outputs are preserved in the public repository to support future comparisons as model capabilities evolve.\u003c/p\u003e \u003cp\u003e Sixth, current guideline formats are not optimized for machine retrieval. Guidelines are designed for human readers rather than computational systems. Development of standards for guideline structuring and maintenance could optimize machine retrievability through standardized section labeling, explicit decision trees, and structured tables encoding treatment criteria. Implementation of continuous corpus updating mechanisms would maintain currency as new evidence emerges and guidelines evolve.\u003c/p\u003e \u003cp\u003eFinally, system transparency and clinician trust remain underexplored. Our evaluation focused on concordance and appropriateness but did not assess clinician trust or usability in real-world settings. Creation of user-centered interfaces exposing retrieval trails would support clinician oversight and trust. Transparency about which guideline sections informed recommendations enables expert validation and iterative refinement.\u003c/p\u003e \u003cp\u003eThese findings suggest that advancing clinical AI requires equal attention to workflow engineering, data curation, and human-AI collaboration as to model development itself. The evaluation framework presented here provides a foundation for rigorous assessment of LLM decision support systems across oncology and other high-stakes clinical domains. By demonstrating both the potential of optimized RAG frameworks to improve concordance and the persistent safety challenges requiring mandatory expert validation, this work establishes a path toward responsible clinical AI deployment that prioritizes patient safety alongside technical performance.\u003c/p\u003e "},{"header":"Methods","content":"\u003cp\u003eStudy Design and Case Selection\u003c/p\u003e\u003cp\u003e This retrospective study was approved by the Institutional Ethics Committee of the University Hospital Tübingen (Protocol No. 273/2024BO1). The cohort included patients discussed at the gastrointestinal MDT of our institution between March 2022 and December 2023. All clinical data were fully anonymized prior to analysis. Patients enrolled in clinical trials were excluded to minimize protocol-driven bias. The study followed STROBE reporting guidelines.\u003c/p\u003e\u003cp\u003e Five tumor subgroups were defined according to German S3 guidelines: esophageal, gastric, pancreatic, hepatobiliary, and colorectal cancer. Using simple random sampling stratified by tumor type and presentation timing, 20 patients per subgroup were selected (10 first presentations and 10 follow-up cases), resulting in 100 cases overall.\u003c/p\u003e\u003cp\u003eClinical data were extracted from standardized MDT templates including patient demographics, tumor characteristics, staging information, prior treatments, and current clinical question. Cases with mixed diagnoses not clearly fitting standard guideline pathways, as well as patients referred to or enrolled in clinical trials, were excluded to create a corpus aligned with guideline-based treatment algorithms.\u003c/p\u003e\u003cp\u003eFramework Architecture\u003c/p\u003e\u003cp\u003eA modular framework was developed to evaluate four components in factorial combination (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e):\u003c/p\u003e\u003col\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003eLLM variant (GPT-4o vs. GPT-4o-mini)\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003eRetrieval interface (custom API-based RAG vs. ChatGPT Assistant)\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003eQuery formulation (original German case notes vs. rewritten structured queries)\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003e Guideline corpus scope (full corpus vs. curated treatment-focused subset)\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003c/ol\u003e\u003cp\u003eThis 2×2×2×2 factorial design resulted in 16 distinct configurations (\u003cem\u003eSupplementary Table S3 in Supplementary Materials\u003c/em\u003e), enabling systematic assessment of how individual components influenced concordance with MDT recommendations.\u003c/p\u003e\u003cp\u003eAll experiments were conducted using the following production snapshots: gpt-4o-2024-11-20 and gpt-4o-mini-2024-07-18.\u003c/p\u003e\u003cp\u003e \u003cstrong\u003eQuery Formulation\u003c/strong\u003e \u003c/p\u003e\u003cp\u003eFor each case, LLMs received structured, anonymized MDT reports containing clinical and oncological variables. Inputs consisted exclusively of structured text extracted from MDT documentation. No imaging files, PDFs, or free-text physician correspondence were provided.\u003c/p\u003e\u003cp\u003e For rewritten-query configurations, an LLM-in-the-loop approach was used, whereby GPT-4o-mini translated German case descriptions into English and restructured the content to align with oncology guideline terminology and format. This step was implemented to standardize case representation across configurations. Informal review of a representative sample of rewritten queries by a bilingual clinician confirmed that clinical meaning and oncological terminology were preserved, though systematic translation quality assessment was beyond the scope of this study. Representative prompt examples are provided in the Supplementary Materials and in the public repository.\u003c/p\u003e\u003cp\u003e \u003cstrong\u003eInformation Retrieval\u003c/strong\u003e \u003c/p\u003e\u003cp\u003eFour retrieval conditions were tested\u003c/p\u003e\u003col\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003eNo retrieval (baseline)\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003eChatGPT Assistant with full guideline PDFs uploaded\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003eCustom RAG using the complete guideline corpus\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003eCustom RAG using a curated corpus\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003c/ol\u003e\u003cp\u003eAll corpora included German S3 guidelines (\u003cem\u003eLeitlinienprogramm Onkologie\u003c/em\u003e) and NCCN Clinical Practice Guidelines in Oncology relevant to each tumor type, current as of January 2024. The curated corpus excluded sections on epidemiology, prevention, screening, and basic diagnostic workup—content judged by two board-certified oncologists to be irrelevant to treatment planning—to reduce retrieval noise.\u003c/p\u003e\u003cp\u003eCustom RAG Implementation\u003c/p\u003e\u003cp\u003e Patient cases and guideline documents were embedded using the multilingual BAAI/bge-m3 model. Guideline PDFs were parsed into hierarchical text structures using pymupdf4llm. Tables were converted to JSON format, and graphical elements were transcribed into text descriptions (\u003cspan class=\"CitationRef\"\u003e11\u003c/span\u003e).\u003c/p\u003e\u003cp\u003eHierarchical indexing was implemented using LlamaIndex. For each query, the top five guideline chunks were retrieved based on cosine similarity in embedding space. This approach aimed to balance focused context retrieval with preservation of clinically relevant information.\u003c/p\u003e\u003cp\u003eThe complete implementation code is available at \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.5281/zenodo.18483659\u003c/span\u003e\u003cspan class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, with detailed documentation of hyperparameters, preprocessing steps, and retrieval algorithms.\u003c/p\u003e\u003cp\u003ePrompting Strategy\u003c/p\u003e\u003cp\u003ePrompts followed the C.R.E.A.T.E. framework (Context, Role, Explicit Instruction, Action, Tone, Examples). The model was instructed to act as a MDT and generate a concise German-language treatment recommendation based on the provided case information, with or without retrieved guideline content.\u003c/p\u003e\u003cp\u003e For retrieval-augmented configurations, retrieved guideline chunks were appended below the case description. Prompts emphasized actionable treatment recommendations and discouraged deferral or requests for additional diagnostics. Representative prompt templates are provided in the Supplementary Material and in the public repository.\u003c/p\u003e\u003cp\u003eOutcome Classification\u003c/p\u003e\u003cp\u003eMDT and LLM-generated recommendations were independently classified by two board clinicians, blinded to both the study framework and the original recommendations, into the following nine therapeutic categories:\u003c/p\u003e\u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eBest supportive care\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eFurther diagnostic procedures (without subsequent therapy plan)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eEndoscopic intervention (e.g., endoscopic resection)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eActive surveillance or follow-up\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eMultimodal therapy (concurrent use of multiple treatment modalities, such as radiochemotherapy)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eMultistep therapy (sequential treatment strategies, e.g., neoadjuvant chemotherapy followed by surgery and eventually adjuvant therapy)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eSurgery (invasive surgical procedures requiring general anesthesia)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eSystemic therapy (including chemotherapy and immunotherapy)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eLocalized therapy (e.g., radiotherapy, radiofrequency ablation, transarterial chemoembolization, or stereotactic body radiotherapy).\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e\u003cp\u003eClassification was guided by the dominant therapeutic intent. When multiple treatment options were documented by the MDT, LLM recommendations were considered concordant if they matched any accepted option. Disagreements between reviewers were resolved through discussion until consensus was reached. This approach reflects the rate of guideline-consistent recommendations rather than strict one-to-one agreement with MDT decisions.\u003c/p\u003e\u003cp\u003eDetailed classification criteria and examples are provided in Supplementary Table S4 in Supplementary Material and in the public repository.\u003c/p\u003e\u003ch2\u003eStatistical Analysis\u003c/h2\u003e\u003cp\u003eConcordance rates were calculated for each framework configuration. Overall differences across configurations were assessed using Cochran's Q test, followed by pairwise McNemar tests with continuity correction (α = 0.05). Subgroup analyses examined tumor type, presentation timing, and treatment category.\u003c/p\u003e\u003cp\u003e Retrieval similarity between configurations was quantified using the Jaccard similarity index, defined as the intersection divided by the union of retrieved guideline chunks.\u003c/p\u003e\u003cp\u003ePower analysis was performed using McNemar's test for paired binary outcomes, assuming asymmetric discordance corresponding to large effect sizes. Sensitivity analyses evaluated power to detect moderate and large differences in concordance with a sample size of 100 cases (see Supplementary Material).\u003c/p\u003e\u003cp\u003eFramework-level concordance per case was classified as “Mostly Wrong” (≤ 30%), “Mostly Correct” (\u0026gt; 90%), or “Fully Concordant” (100%), based on predefined thresholds.\u003c/p\u003e\u003cp\u003e We quantified the number of retrieved guideline chunks containing graphical elements (e.g. figures and flowcharts) for each retrieval-augmented configuration by automated detection of image and table tags in parsed PDF content.\u003c/p\u003e\u003cp\u003eEvaluation of Discordant Recommendations\u003c/p\u003e\u003cp\u003eTo assess clinical appropriateness of incorrect AI-generated recommendations, we conducted expert evaluation focusing on discordant cases from the best-performing configuration for both model variants. Each recommendation was independently reviewed by two board-certified oncologists who were blinded to model identity. Reviewers classified each recommendation as:\u003c/p\u003e\u003col\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003e\"Off-base\"—medically inappropriate or not clinically justifiable based on current guidelines and patient characteristics\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003e \"Alternative\"—differing from MDT recommendation but representing a potentially acceptable therapeutic option supported by guidelines or clinical evidence\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003c/ol\u003e"},{"header":"Abbreviations","content":"\u003cdiv class=\"DefinitionList\"\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eAI\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eArtificial Intelligence\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eHCC\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eHepatocellular Carcinoma\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eLLM\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eLarge Language Model\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eMDT\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eMultidisciplinary Tumor Board\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eRAG\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eRetrieval-augmented generation\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eFunding:\u003c/strong\u003e no funding was received.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePrior presentation:\u003c/strong\u003e none.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDisclaimer:\u003c/strong\u003e none.\u003c/p\u003e\n\u003cp\u003eEthics approval and consent to participate\u003c/p\u003e\n\u003cp\u003eThis retrospective study was approved by the Institutional Ethics Committee of the University Hospital T\u0026uuml;bingen (Protocol No. 273/2024BO1).\u003c/p\u003e\n\u003cp\u003eThe requirement for informed consent was waived by the Ethics Committee due to the retrospective and anonymized nature of the study.\u003c/p\u003e\n\u003cp\u003eData Availability\u003c/p\u003e\n\u003cp\u003ePatient-level clinical data analysed in this study are subject to institutional data protection policies and cannot be made publicly available. The original patient cases and full MDT recommendations therefore remain restricted clinical datasets. Access to non-anonymized data may be considered on reasonable request, subject to institutional approval, applicable data use agreements, and secure data transfer mechanisms.\u003c/p\u003e\n\u003cp\u003eTo support transparency and reproducibility, the authors provide a public repository containing a fully anonymized study dataset and all analysis code. The dataset includes anonymized MDT treatment recommendations, the raw responses generated by the large language models for all 16 experimental frameworks, derived treatment classes, binary concordance labels, and retrieval metadata (retrieved chunk indices and chart counts) enabling replication of concordance and retrieval analyses. Together with the accompanying scripts, these materials allow full methodological reproduction of the study while respecting patient privacy and licensing constraints.\u003c/p\u003e\n\u003cp\u003eThe repository is archived with a persistent DOI: \u003cem\u003ehttps://doi.org/10.5281/zenodo.18483659\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eCode Availability\u003c/p\u003e\n\u003cp\u003eComplete source code, documentation, and detailed implementation specifications are publicly available in the same repository (\u003cem\u003ehttps://doi.org/10.5281/zenodo.18483659\u003c/em\u003e). The repository includes all preprocessing scripts, embedding and retrieval pipelines, prompt templates, and analysis scripts required to reproduce the experimental setup and the reported results. No proprietary software is required beyond standard Python libraries and publicly documented APIs.\u003c/p\u003e\n\u003cp\u003eAcknowledgments\u003c/p\u003e\n\u003cp\u003eThis work received no external funding. The authors acknowledge support from their respective departments and institutions.\u003c/p\u003e\n\u003cp\u003eAuthor Contributions\u003c/p\u003e\n\u003cp\u003eIC and CE conceived and designed the study. DS and ADP performed data collection and curation. DS, ADP, and IC developed the technical framework and conducted experiments. PM and MB performed expert evaluation of discordant cases. ALM and MB provided clinical oversight and guideline expertise. DS and IC performed statistical analyses. DS and IC drafted the manuscript. All authors critically revised the manuscript and approved the final version.\u003c/p\u003e\n\u003cp\u003eCompeting Interests\u003c/p\u003e\n\u003cp\u003eThe authors declare no competing financial or non-financial interests.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eBraulke F, Kober K, Arndt A, Papendick M, Strauss A, Kramm CM, et al. Optimizing the structure of interdisciplinary tumor boards for effective cancer care. Front Oncol. 2023;13.\u003c/li\u003e\n\u003cli\u003eChau BL, LaGuardia JS, Kim S, Zhang SC, Pletcher E, Sanford NN, et al. Association of Parental Status and Gender With Burden of Multidisciplinary Tumor Boards Among Oncology Physicians. Jama Network Open. 2023;6(10).\u003c/li\u003e\n\u003cli\u003eCone EB, Marchese M, Paciotti M, Nguyen DD, Nabi J, Cole AP, et al. Assessment of Time-to-Treatment Initiation and Survival in a Cohort of Patients With Common Cancers. Jama Network Open. 2020;3(12).\u003c/li\u003e\n\u003cli\u003eOureilidis-DeVivo B. The physician oncology referral process and its effect on tumor board outcomes. J Clin Oncol. 2024;42(16).\u003c/li\u003e\n\u003cli\u003eErdat EC, Yal\u0026ccedil;iner M, \u0026Ouml;r\u0026uuml;nc\u0026uuml; MB, \u0026Uuml;r\u0026uuml;n Y, Senler F\u0026Ccedil;. Assessing the accuracy of the GPT-4 model in multidisciplinary tumor board decision prediction. Clin Transl Oncol. 2025.\u003c/li\u003e\n\u003cli\u003eAghamaliyev U, Karimbayli J, Giessen-Jung C, Matthias I, Unger K, Andrade D, et al. ChatGPT\u0026apos;s Gastrointestinal Tumor Board Tango: A limping dance partner? Eur J Cancer. 2024;205.\u003c/li\u003e\n\u003cli\u003eLiao N, Li CK, Gradishar WJ, Klimberg VS, Roshal JA, Yuan TZ, et al. Accuracy and Reproducibility of ChatGPT Responses to Breast Cancer Tumor Board Patients. Jco Clin Cancer Info. 2025;9.\u003c/li\u003e\n\u003cli\u003eLevin G, Gotlieb W, Ramirez P, Meyer R, Brezinov Y. ChatGPT in a gynaecologic oncology multidisciplinary team tumour board: A feasibility study. Bjog-Int J Obstet Gy. 2025;132(1):99\u0026ndash;101.\u003c/li\u003e\n\u003cli\u003eBerman E, Sundberg Malek H, Bitzer M, Malek N, Eickhoff C. Retrieval Augmented Therapy Suggestion for Molecular Tumor Boards: Algorithmic Development and Validation Study. J Med Internet Res. 2025;27:e64364.\u003c/li\u003e\n\u003cli\u003eZhang G, Xu Z, Jin Q, Chen F, Fang Y, Liu Y, et al. Leveraging long context in retrieval augmented language models for medical question answering. NPJ Digit Med. 2025;8(1):239.\u003c/li\u003e\n\u003cli\u003eKresevic S, Giuffre M, Ajcevic M, Accardo A, Croce LS, Shung DL. Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework. NPJ Digit Med. 2024;7(1):102.\u003c/li\u003e\n\u003cli\u003eJeong M, Sohn J, Sung M, Kang J. Improving medical reasoning through retrieval and self-reflection with retrieval-augmented large language models. Bioinformatics. 2024;40(Suppl 1):i119\u0026ndash;i29.\u003c/li\u003e\n\u003cli\u003eWang CL, Ong J, Wang CR, Ong H, Cheng RBK, Ong D. Potential for GPT Technology to Optimize Future Clinical Decision-Making Using Retrieval-Augmented Generation. Ann Biomed Eng. 2024;52(5):1115\u0026ndash;8.\u003c/li\u003e\n\u003cli\u003eAmugongo LM, Mascheroni P, Brooks S, Doering S, Seidel J. Retrieval augmented generation for large language models in healthcare: A systematic review. PLOS Digit Health. 2025;4(6):e0000877.\u003c/li\u003e\n\u003cli\u003eNeha F, Bhati D, Shukla DK. Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review. Ai-Basel. 2025;6(9).\u003c/li\u003e\n\u003cli\u003eVach M, Gliem M, Weiss D, Ivan VL, Hauke F, Boschenriedter C, et al. Evaluating Retrieval Augmented Generation-enhanced Large Language Models for Question Answering On German Neurovascular Guidelines. Clin Neuroradiol. 2025.\u003c/li\u003e\n\u003cli\u003eMasanneck L, Meuth SG, Pawlitzki M. Evaluating base and retrieval augmented LLMs with document or online support for evidence based neurology. Npj Digital Medicine. 2025;8(1).\u003c/li\u003e\n\u003cli\u003eWolk K. Evaluating Retrieval-Augmented Generation Variants for Clinical Decision Support: Hallucination Mitigation and Secure On-Premises Deployment. Electronics-Switz. 2025;14(21).\u003c/li\u003e\n\u003cli\u003eShah K, Xu AY, Sharma Y, Daher M, McDonald C, Diebo BG, et al. Large Language Model Prompting Techniques for Advancement in Clinical Medicine. J Clin Med. 2024;13(17).\u003c/li\u003e\n\u003cli\u003eHuo B, Boyle A, Marfo N, Tangamornsuksan W, Steen JP, McKechnie T, et al. Large Language Models for Chatbot Health Advice Studies: A Systematic Review. JAMA Netw Open. 2025;8(2):e2457879.\u003c/li\u003e\n\u003cli\u003eKim TT, Makutonin M, Sirous R, Javan R. Optimizing Large Language Models in Radiology and Mitigating Pitfalls: Prompt Engineering and Fine-tuning. Radiographics. 2025;45(4):e240073.\u003c/li\u003e\n\u003cli\u003eMyers S, Miller TA, Gao Y, Churpek MM, Mayampurath A, Dligach D, et al. Lessons learned on information retrieval in electronic health records: a comparison of embedding models and pooling strategies. J Am Med Inform Assoc. 2025;32(2):357\u0026ndash;64.\u003c/li\u003e\n\u003cli\u003eEmssaad I, Ben-Bouazza FE, Tafala I, Chakour El Mezali M, Jioudi B. Leveraging multilingual RAG for breast cancer RCPs: AI-driven speech transcription and compliance in Darija-French clinical discussions. Computer Methods and Programs in Biomedicine Update. 2025;8.\u003c/li\u003e\n\u003cli\u003eYang R, Ning Y, Keppo E, Liu M, Hong C, Bitterman DS, et al. Retrieval-augmented generation for generative artificial intelligence in health care. npj Health Syst. 2025;2.\u003c/li\u003e\n\u003cli\u003eN\u0026eacute;v\u0026eacute;ol A, Dalianis H, Velupillai S, Savova G, Zweigenbaum P. Clinical Natural Language Processing in languages other than English: opportunities and challenges. J Biomed Semant. 2018;9.\u003c/li\u003e\n\u003cli\u003eGaschi F, Fontaine X, Rastin P, Y. T, editors. Multilingual Clinical NER: Translation or Cross-lingual Transfer? Proceedings of the 5th Clinical Natural Language Processing Workshop; 2023; Toronto, Canada: Association for Computational Linguistics.\u003c/li\u003e\n\u003cli\u003eBenary M, Wang XD, Schmidt M, Soll D, Hilfenhaus G, Nassir M, et al. Leveraging Large Language Models for Decision Support in Personalized Oncology. JAMA Netw Open. 2023;6(11):e2343689.\u003c/li\u003e\n\u003cli\u003eChen D, Avison K, Alnassar S, Huang RS, Raman S. Medical accuracy of artificial intelligence chatbots in oncology: a scoping review. Oncologist. 2025;30(4).\u003c/li\u003e\n\u003cli\u003eLammert J, Dreyer T, Mathes S, Kuligin L, Borm KJ, Schatz UA, et al. Expert-Guided Large Language Models for Clinical Decision Support in Precision Oncology. JCO Precis Oncol. 2024;8:e2400478.\u003c/li\u003e\n\u003cli\u003eLammert J, Dreyer TF, L\u0026ouml;rsch AM, Jung J, Lange S, Pfarr N, et al. Large language models for precision oncology: Clinical decision support through expert-guided learning. J Clin Oncol. 2024;42(16).\u003c/li\u003e\n\u003cli\u003eIzacard G, Grave E. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. 16th Conference of the European Chapter of the Association for Computational Linguistics (Eacl 2021). 2021:874\u0026ndash;80.\u003c/li\u003e\n\u003cli\u003eLi M, Kilicoglu H, Xu H, Zhang R. BiomedRAG: A retrieval augmented large language model for biomedicine. J Biomed Inform. 2025;162:104769.\u003c/li\u003e\n\u003cli\u003ePoretsky E, Blake VC, Andorf CM, Sen TZ. Assessing the performance of generative artificial intelligence in retrieving information against manually curated genetic and genomic data. Database (Oxford). 2025;2025.\u003c/li\u003e\n\u003cli\u003eLiu S, McCoy AB, Wright A. Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. J Am Med Inform Assoc. 2025;32(4):605\u0026ndash;15.\u003c/li\u003e\n\u003cli\u003eMenz BD, Kuderer NM, Bacchi S, Modi ND, Chin-Yee B, Hu T, et al. Current safeguards, risk mitigation, and transparency measures of large language models against the generation of health disinformation: repeated cross sectional analysis. BMJ. 2024;384:e078538.\u003c/li\u003e\n\u003cli\u003eCarl N, Schramm F, Haggenmuller S, Kather JN, Hetz MJ, Wies C, et al. Large language model use in clinical oncology. NPJ Precis Oncol. 2024;8(1):240.\u003c/li\u003e\n\u003cli\u003eSharma P, Hassan C. Artificial Intelligence and Deep Learning for Upper Gastrointestinal Neoplasia. Gastroenterology. 2022;162(4):1056\u0026ndash;66.\u003c/li\u003e\n\u003cli\u003eChen LC, Zack T, Demirci A, Sushil M, Miao B, Kasap C, et al. Assessing Large Language Models for Oncology Data Inference From Radiology Reports. JCO Clin Cancer Inform. 2024;8:e2400126.\u003c/li\u003e\n\u003cli\u003eMaida M, Celsa C, Lau LHS, Ligresti D, Baraldo S, Ramai D, et al. The Application of Large Language Models in Gastroenterology: A Review of the Literature. Cancers (Basel). 2024;16(19).\u003c/li\u003e\n\u003cli\u003eShah MA, Kennedy EB, Catenacci DV, Deighton DC, Goodman KA, Malhotra NK, et al. Treatment of Locally Advanced Esophageal Carcinoma: ASCO Guideline. J Clin Oncol. 2020;38(23):2677\u0026ndash;94.\u003c/li\u003e\n\u003cli\u003eLiang S, Zhang J, Liu X, Huang Y, Shao J, Liu X, et al. The potential of large language models to advance precision oncology. EBioMedicine. 2025;115:105695.\u003c/li\u003e\n\u003cli\u003eShool S, Adimi S, Saboori Amleshi R, Bitaraf E, Golpira R, Tara M. A systematic review of large language model (LLM) evaluations in clinical medicine. BMC Med Inform Decis Mak. 2025;25(1):117.\u003c/li\u003e\n\u003cli\u003eLewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Adv Neur In. 2020;33.\u003c/li\u003e\n\u003cli\u003eKaiser KN, Hughes AJ, Yang AD, Turk AA, Mohanty S, Gonzalez AA, et al. Accuracy and consistency of publicly available Large Language Models as clinical decision support tools for the management of colon cancer. J Surg Oncol. 2024;130(5):1104\u0026ndash;10.\u003c/li\u003e\n\u003cli\u003eCapobianco I, Della Penna A, Mihaljevic AL, Bitzer M, Eickhoff C, Stifini D. Clinical Accuracy and Safety Concerns Following GPT-5 Public Demonstration in Cancer Care. J Med Syst. 2025;49(1).\u003c/li\u003e\n\u003cli\u003eCan E, Uller W, Kotter E, Vogt K, Doppler M, Bronnimann M, et al. Comparative Evaluation of Proprietary and Open-Source Large Language Models for Systematic Multi-source Information Extraction in Interventional Oncology. Cardiovasc Intervent Radiol. 2025.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-8849187/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8849187/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eLarge language models (LLMs) have been proposed as decision support tools for multidisciplinary tumor boards, yet systematic preclinical validation of retrieval-augmented generation (RAG) pipelines remains lacking. In this retrospective framework validation study using real-world clinical data, we applied a modular evaluation framework to 100 gastrointestinal tumor board cases spanning five cancer types, systematically testing 16 configurations varying model variant, multilingual retrieval strategy, query formulation, and corpus scope.\u003c/p\u003e \u003cp\u003eBaseline concordance with multidisciplinary team recommendations ranged from 79\u0026ndash;85%. Combining query rewriting with curated guideline retrieval improved concordance to 93\u0026ndash;95% (p\u0026thinsp;\u0026lt;\u0026thinsp;0.01), with prompt design and corpus curation exerting greater influence than model selection. Among residual discordant cases in optimal configurations, approximately 60% represented clinically inappropriate recommendations rather than acceptable therapeutic alternatives.\u003c/p\u003e \u003cp\u003eThese findings demonstrate that systematic RAG optimization substantially improves clinical decision support concordance, while the high rate of inappropriate residual errors underscores the necessity of mandatory expert oversight before any clinical deployment.\u003c/p\u003e","manuscriptTitle":"Systematic Evaluation of Multilingual Retrieval-Augmented Generation for Gastrointestinal Tumor Board Decision Support","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-08 17:34:02","doi":"10.21203/rs.3.rs-8849187/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-04-08T20:55:44+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-08T01:54:24+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-07T20:59:02+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-03T22:00:52+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-01T14:04:37+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-01T08:00:41+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-03-29T09:32:25+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"292690696025120281209602257785885649975","date":"2026-03-10T17:39:10+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"56659618095500670632560855220490095045","date":"2026-03-09T16:16:59+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"215732400753542853153202267617746087448","date":"2026-03-08T02:31:06+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"60891027930169116061862315090095860118","date":"2026-03-05T01:43:48+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"8545691846565819306075574466694547432","date":"2026-03-03T09:24:40+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"120601690732109193742931161843942115169","date":"2026-03-03T05:55:03+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"241705357540545018961788311029306991781","date":"2026-03-02T23:12:13+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-03-02T22:46:03+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-02-13T20:04:02+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-02-13T12:18:12+00:00","index":"","fulltext":""},{"type":"submitted","content":"npj Digital Medicine","date":"2026-02-11T08:12:03+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"11e28a27-6a15-4899-9ca8-8aa7a5cfbae0","owner":[],"postedDate":"March 8th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":63946392,"name":"Biological sciences/Cancer"},{"id":63946393,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":63946394,"name":"Health sciences/Oncology"}],"tags":[],"updatedAt":"2026-04-21T14:25:00+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-08 17:34:02","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8849187","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8849187","identity":"rs-8849187","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}