Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation

Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation Cheng-Yi Li, Kao-Jung Chang, Cheng-Fu Yang, Hsin-Yu Wu, Wenting Chen, and 8 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4558754/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 06 Mar, 2025 Read the published version in Nature Communications → Version 1 posted You are reading this latest preprint version Abstract Multi-modal large language models (MLLMs) have been given free rein to explore exciting medical applications with a primary focus on radiology report generation. Nevertheless, the preliminary MLLM successful attempts in 2D medical image-text pair captioning are incompetent to reflect the real-world diagnostic challenge in the volumetric 3D anatomy. Toward deploying MLLM for more applicable diagnostic context, we noticed that the (1) scarcity of 3D image training dataset, (2) the direct use of undifferentiated foundation MLLMs, and (3) the lack of pertinent caption evaluation metrics were independent domain-specific constraints that integratively hobbles the iteration of next-generation medical MLLM research. In this regard, this study collected a 3D-BrainCT dataset (18,885 text-scan pairs) and applied clinical visual instruction tuning (CVIT) to train volumetric anatomy-sensible BrainGPT models to generate radiology-adherent 3D brain CT reports. Statistically, our BrainGPT model scored BLEU-1 = 44.35, BLEU-4 = 20.38, METEOR = 30.13, ROUGE-L = 47.6, and CIDEr-R = 211.77 during internal testing and demonstrated an accuracy of 0.91 in captioning midline shifts on the external validation CQ500 dataset. By further inspecting the captioned report, we reported that the traditional metrics appeared to measure only the surface text similarity and failed to gauge the information density of the diagnostic purpose. To close this gap, we proposed a novel Feature-Oriented Radiology Task Evaluation (FORTE) to estimate the clinical relevance (lesion feature and landmarks) of the report. Notably, the BrainGPT model scored an average FORTE 0.71 F1-score (degree=0.661; landmark=0.706; feature=0.693, and impression=0.779). To demonstrate that BrainGPT models possess objective readiness to generate human-like radiology reports, we conducted a Turing test that enrolled 11 physician evaluators, and around 74% of the BrainGPT-generated captions were indistinguishable from those written by humans. While various computational intelligence researchers have advocated the avant-garde MLLM applications, our work embodies a holistic framework that showcased the first-hand experience of curating a 3D brain CT dataset, fine-tuning anatomy-sensible language models, and proposing robust radiology evaluation metrics. We deemed that the adventure of docking MLLM for 3D brain CT report generation may unfold new MLLM applications at the forefront of human-machine collaborated modern healthcare. Biological sciences/Computational biology and bioinformatics/Machine learning Health sciences/Health care/Medical imaging/Three-dimensional imaging Multi-modal Large Language Model (MLLM) Three-dimensional (3D) brain computed tomography (CT) Visual instruction tuning Turing test Clinical Efficacy (CE) Evaluation Metric Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Main The rapid ascent of artificial intelligence (AI) technologies has profoundly influenced the interpretation of multimodal information such as computed tomography (CT), 1 gross dermatologic conditions, 2 cytology slides, 3 ophthalmology images, 4 etc. While modern models excel in tasks like classification, regression, and segmentation, they fall short in providing the natural-language expressivity and interactivity required in hands-on clinical workflows. 5 This discrepancy is critical because instant scan interpretation is essential to clinical decision-making, with the rate decision step being the radiologist's report writing. 6,7 To address this need, early generative models. 8–11 have been applied to chest X-ray (CXR) report generation in datasets such as the MIMIC-CXR and OpenI to relieve radiologists' time-consuming and tedious burden. 12,13 These studies demonstrated a potential interface where human-computer collaboration can form medical reports, which can be further refined with multi-modal large language models (MLLM). 14 To examine the full adequacy of MLLM in radiology report generation, we conducted a thorough review of relative works. Despite the primary success in CXR report generation, we identified three crucial limitations in the existing literature, including ( 1 ) data complexity, ( 2 ) model capacity, and ( 3 ) evaluation metric fidelity. These innate limitations have hindered the development of totipotent MLLMs. While the human body is inherently three-dimensional (3D), existing 2D CXR image datasets 12,13 have yet to encompass ( 1 ) pathologic features in complex neuro-vascular anatomy, i.e., the brain, heart, and eye, ( 2 ) spatial landmarks within the 3D image context, and ( 3 ) the intensity degree or size of lesions. Moreover, the single-slice image content from current CT image-text data is usually selected from 3D data to include the described feature. This may be subject to sharpshooter fallacy , where the selected slice often indicates the caption target of interest. Hence, it is imperative to curate a 3D image set to inspect the dexterities of MLLMs in real-world diagnostic contexts. Crucially, MLLMs must be capable of multi-image reasoning or video understanding when used to generate reports for real-world 3D data. Recent medical MLLM studies such as LLaVA-Med (Microsoft) and Med-PaLM M (Google Research) have demonstrated promising X-ray and single-slice CT report generation performance. 15,16 However, during testing, the brain CT data is presented in a 3D format, which poses challenges for the MLLM to identify the specific slice containing the lesion of interest precisely in situ . 3D CT scans of the brain, with reports structured in a list-by-list format focusing on differential diagnoses, serve as the first-line diagnostic for diverse intracranial conditions. 17,18 The brain lesion's degree, size, and location are paramount to forming precise diagnoses and guiding subsequent clinical decisions. Therefore, the value of key radiology descriptors outweighs grammatical filler words. In this regard, traditional evaluation metrics, designed to evaluate short translation 19,20 , abstract summary tasks 21 , and common image captioning 22 cannot gauge the clinical essence of Brain CT reports. 23 On the other hand, the CheXpert clinical efficacy evaluation is designed to capture pathology keyword retrieval rates to represent the CXR report quality. 11,24 However, 14-feature retrieval rates cannot reflect the multi-semantic context (inclusive or exclusive; progression or resolution; relative location and size) of disease features. Moreover, the uncategorized keyword bank concept may not be transferable from one image modality to other modalities. To alleviate such constraints, radiology semantic categories must be incorporated into a novel scoring system to measure the clinical essence of radiology reports. In this study, we aim to advance the application of MLLMs in radiology by addressing several critical areas: ( 1 ) We curated a large-scale 3D-brainCT dataset (18,885 text-scan pairs) abundant with lesion degree, spatial landmarks, and diagnostic impressions of neuronal and vascular features to present a real-world challenge for MLLMs. (Fig. 1 and Extended Data Table 1) ( 2 ) We applied the clinical visual instruction tuning (CVIT) concept to train the BrainGPT model from the open-source Otter. 25,26 The Otter model’s multi-image captioning capacity and clinical instruction design enable the generation of clinically sensible captions from volumetric brain CT scans. (Fig. 2 ) The BrainGPT was externally validated on CQ500 (Fig. 5 ) and an 11-physician rater Turing Test-like linguistic-embedded evaluation. (Fig. 6 ) ( 3 ) We proposed feature-oriented radiology task evaluation (FORTE) to measure the clinical relevance of captions based on distinct disease objectives, including 4 generalizable keyword categories (degree, landmark, feature, and impression). (Fig. 4 and Extended Data Fig. 2 ) We also suggested that preprocessing processes, such as sentence pairing and negation removal, are beneficial in adapting traditional metrics to the detailed, itemized, and differential diagnosis-oriented format of brain CT reports. (Figs. 3 and 4 c) Overall, this study proposes a holistic framework that addresses the implementation details of applying MLLMs in brain CT and other complex anatomy images. Results Training BrainGPT with Clinical Visual Instruction Tuning To implement medical MLLM in 3D brain CT report generation, we collected a real-world 3D-BrainCT dataset and trained BrainGPT models by applying visual instruction tuning on the Otter foundation model (Extended Data Fig. 4 ). Our fine-tuning conditions included regular visual instruction tuning (RVIT): ( 1 ) Plain Instruction (conveys the model’s role as a radiology assistant) and ( 2 ) In-context example Instruction 27 (3-shot examples added to the Plain Instruction), or clinical visual instruction tuning (CVIT): ( 3 ) Template Instruction (structured clinical-defined QA templates added to the Plain Instruction) and ( 4 ) Keyword Instruction (categorical guidelines focused on keywords added to the Plain Instruction). As a result, we derived four BrainGPT models (-plain, -example, -template, -keyword) that are tuned to perform CT interpretation with different prior levels of clinical sensibility (Fig. 2 ). 25 At the whole report level, all of the four fine-tuned BrainGPT models outperformed the baseline Otter model in traditional metrics (Mann-Whitney U test p < 0.01, see Fig. 3 b). In particular, the baseline Otter model scored the lowest in BLEU-4 (0) and CIDEr-R (5.9) metrics, indicating that the baseline model-generated CT report had low n-gram overlap and underfit semantic term usage frequency. In addition, we observed a mismatch between the conventional metric and the report quality by reviewing generated reports (see Extended Data Table 1). Moreover, no traditional metrics, except CIDEr-R, rated the BrainGPT performance in accord with the crescendo elaborateness of RVIT and CVIT (Mann-Whitney U test p > 0.05 across four visual instruction tuning conditions, summarized in Extended Data Table 2). Therefore, we demonstrated that traditional metrics were inherently insensitive to the clinical essence of the generated radiology reports. Sentence pairing To investigate whether the list-by-list report architecture of brain CT images accounted for low traditional metric scores, we applied sentence pairing to decompose the multi-sentence paragraph into smaller semantic granularity. (Fig. 3 a) As a result, the process of sentence pairing increased traditional metric scores by an average of 5.28 points in METEOR, 6.48 points in ROUGE-L, and an astonishing 114 points in CIDEr-R (Fig. 3 b and Extended Data Table 2). The traditional metric scores for every BrainGPT model were significantly improved across most traditional metrics ( p 0.7) in BLEU-2, BLEU-3, BLEU-4, METEOR, and ROUGE-L with statistical significance ( p -value < 0.05). However, apart from the other n-gram-based traditional metric, the CIDEr-R score gained a more positive shift from the process of sentence pairing (r = 0.577). Moreover, we observed a positive correlation between advanced CVIT and increased traditional metric scores. In particular, the CIDEr-R score depicted the most prominent ascending trend across the hierarchically fine-tuned BrainGPT-plain (125.86), BrainGPT-example (132.38), BrainGPT-template (147.92), and BrainGPT-keyword (153.3). This sentence pairing procedure not only relieved the sequential constraint between the report input and generated output but also accentuated the clinical value in the list-by-list differential diagnosis reports. Feature-Oriented Radiology Task Evaluation (FORTE) Given that CIDEr-R captures the hierarchical clinical essence across visual instruction tuning conditions, we hypothesized that its frequency-inverse document frequency (TF-IDF) component reacts to the usage of rare radiology keywords in reports. Therefore, we analyzed the term frequency in both the ground truth and test outputs (Fig. 4a 1 ). We found that the baseline Otter model had a low recall rate for keywords, whereas the BrainGPT models, after visual instruction tuning, demonstrated significant radiology keyword usage in image captioning. Notably, the BrainGPT-keyword (Fig. 4a 2 : y = 0.95x-1.18; r 2 = 0.85) maintained a high recall rate across varying frequencies of radiology keywords. Due to the potential instability caused by keyword frequency variance in report evaluations, we introduced the feature-oriented radiology task evaluation (FORTE). This method gauges the medical content of the generated report by focusing on the density of radiology information. We categorized radiology keywords into degree, landmark, feature, and impression subsets to offer a multi-faceted evaluation of the system's performance. By evaluating the F1 score of each FORTE-measured keyword category (see Extended Data Table 4), we noted that the advanced CVIT models, BrainGPT-template and BrainGPT-keyword, scored higher average F1 score than the RVIT models. (Mann-Whitney U test p -value < 0.001, as detailed in Extended Data Table 5) Specifically, BrainGPT-keyword captured the most radiology details, with FORTE F1 scores of: degree = 0.548, landmark = 0.533, feature = 0.574, and impression = 0.649. These collective results indicate that clinical instruction-tuned BrainGPT models generate brain CT reports with a higher level of radiology term usage and better alignment with original diagnostic reports. Compare FORTE against traditional evaluation metrics To determine if FORTE reveals report qualities uncovered by traditional metrics, we conducted a Pearson correlation coefficient analysis to compare FORTE with traditional metrics (Fig. 4 d). We observed that traditional metrics had high intra-correlations (r > 0.7, p < 0.001), but lower correlation with FORTE (r < 0.5, p < 0.001) with the FORTE. In addition, the four FORTE domains (degree, landmark, feature, and impression) had lower intra-correlations (r < 0.5, p < 0.001 compared to traditional metrics). This comparison indicated that the FORTE addresses broader and distinct disease aspects than those traditional metrics depicted. Negation Removal: Effect of “Interpretation Spree” Upon further review of keyword usage frequency, we found that “no” was notably among the most frequently used words, along with “of,” “and,” “in,” and “the.” This contradicts the common belief of “reporting bias,” where it is assumed that language models, similar to laypersons, are less likely to mention negative features. Clinically, radiologists often focus on context-oriented descriptions to “rule out” diagnostic targets, a practice not mirrored by BrainGPT. Consequently, BrainGPT’s reports were overlaid with excessive negative descriptions rather than the concise, context-specific language used by radiologists, a phenomenon we termed the “Interpretation spree.” To address this, we applied negation removal to limit descriptions to positive findings. (Fig. 4 c) The average score increases were: BLEU-1 = 29.25%, BLEU-2 = 34.9%, BLEU3 = 50.54%, BLEU-4 = 57.26%, METEOR = 24.97%, ROUGE_L = 29.04%, CIDEr-R = 46.6%, with statistical significance (See Extended Data Fig. 1 ). Additionally, negation removal improved the BrainGPT FORTE keyword average F1 score by 0.118 for degree, 0.194 for landmark, 0.139 for feature, and 0.163 for impression (see Fig. 4 b and Extended Data Table 4). Negation removal led to an overall average F1 score gain of 0.153, with the best instruction-tuned model, BrainGPT-keyword, seeing its F1 score rise from 0.576 to 0.71. Together, we demonstrated that negation removal allowed evaluation metrics to focus on positive findings. Meanwhile, negation removal also enhanced average traditional scores and FORTE F1 scores. Assess BrainGPT Generalization by CQ500 External Validation We conducted zero-shot external validation on the CQ500 brain CT dataset (n = 133) to assess BrainGPT's ability to perform report captioning in generalized clinical conditions. We estimated the clinical keyword retrieval rate by comparing BrainGPT's reports to the ground truth labels of the CQ500 CT scans (Fig. 5 a). Notably, BrainGPT mentioned keywords (e.g., ventricular enlargement, atrophy, infarction, and mass effect) in the generated reports for the CQ500 validation dataset with frequencies similar to the training dataset (3D-brainCT). These zero-shot captioning results indicated that BrainGPT had inscribed the CT report phraseology and writing structure for brain disease differential diagnosis. CQ500 is an intracranial hemorrhage (ICH) dataset encompassing “mass effect,” “hemorrhage,” and “midline shift” features, which were less indexed in our geriatric population 3D-brainCT dataset. Nevertheless, the four instruction-tuned BrainGPT models achieved fair accuracy in reporting mass effect (0.71–0.75), midline shift (0.35–0.38), and hemorrhagic events (0.43–0.5). Furthermore, consistent with our previous findings, negation removal (aligning with differential diagnosis interest) significantly increased report accuracy for midline shift (0.86–0.91) and hemorrhagic events (0.59–0.61). Notably, in terms of diagnosing “mass effect,” “hemorrhage,” and “midline shift,” the advanced CVIT models (BrainGPT-template and BrainGPT-keyword) generally outperformed the RVIT fine-tuned models (BrainGPT-plain and BrainGPT-example). This suggests that CVIT instructions acted as a pre-model rectifier, enhancing the focus on valuable clinical etiologies. To further investigate diagnostic credibility, we examined report results for lacunar infarct, subdural hemorrhage (SDH), subarachnoid hemorrhage (SAH), and midline shift with mass effect (Fig. 5 b). Interestingly, in the lacunar infarction case, BrainGPT-example accurately identified the hypodense old lacunar infarct in the putamen and thalamus regions. However, “putamen” was misspelled as “putmen,” likely due to a typo learned from the original training report. The model also generated successful captions for brain lesions, including SDH at the right occipital, SAH, and left uncal herniation. To examine if BrainGPT was subject to the sharpshooter fallacy in 3D CT scans, we isolated multi-object cases with different lesion features in various CT cuts (Fig. 5 c). We noted that in some multi-lesional brain CT, the BrainGPT reports could recognize distinct hemorrhage subtypes (contusion, intraventricular and subarachnoid hemorrhage) with a sparse comprehension of the lesion location (frontotemporal lobe). In another case, BrainGPT described a regressing chronic ICH and a periventricular angiopathic leukoencephalopathy with white matter hypo-density. These multi-object, multi-slice caption reports demonstrated BrainGPT's potential to provide differential diagnoses for hemorrhages and hypodense image features. Linguistic-embedded Turing test Our results demonstrated that the BrainGPT models were capable of emulating the writing style of radiologists, using domain-specific language, and providing preliminary diagnostic information. To further evaluate whether these BrainGPT-generated reports could be distinguished by healthcare providers, we conducted a Turing test. (For the questionnaire design, see Extended Data Fig. 5 ) In this test, we enrolled 11 physicians (2 radiologists, 2 neurologists, and 7 other licensed medical doctors) (Fig. 6 a). These doctors were asked to determine whether 6 provided report cases (two lacunar infarcts, cortex atrophy, subdural hemorrhage, and midline shift) were written by machines or humans. Interestingly, out of 66 evaluations, 74.24% of BrainGPT-generated captions were mistakenly identified as human-written, whereas only 46.97% of human-written reports were correctly identified. When we provided the evaluators with the input CT images, nearly 18% of their decisions were changed for BrainGPT reports, resulting in 56% of machine-written reports being indistinguishable from human-written reports. This input image-directed decision change was not observed in real human-written reports, which had a 50% correct identification rate. Overall, integrating the scan-text context into the Turing test led to more confident evaluations on the 5-point Likert scale ( p -value). This suggests that reviewing actual image contexts helps human evaluators detect subtle differences between machine and human-written reports, likely due to off-target descriptions and inadequate precision in machine-generated captions. We also analyzed the qualitative traits influencing evaluator decisions (Fig. 6 b). We found that the traits of “Familiarity and voice” (29%), "Specificity or vagueness of details" (37.60%), “Continuity and Coherence” (33.33%), and “Writing quality at sentence level” (33.33%) were significant factors in the evaluators' decisions. 28 Notably, when reviewers considered "Familiarity and voice," the average success rate increased by 73.7% (from 22.22–38.6%). Similarly, considering "Specificity or vagueness of details" increased the average success rate by 47% (from 26.67–39.22%). These linguistic criteria underscore the importance of fine-tuning the radiologist writing style and incorporating radiology-specific details in medical report generation. Discussion In this study, we assembled a holistic framework to address various unmet needs in achieving robust report generation for 3D brain CT. Based on our collective experiences, we recommend applying the following sequential protocol (or an equivalent process) to ensure caption and evaluation rigor: CVIT: Incorporate clinical domain knowledge in implementing visual instruction tuning Combine “sentence pairing” and “negation removal”: Adapt traditional metrics to radiology reports Employ FORTE: Gauge the clinical essence of caption results in multiple aspects Conduct linguistic-embedded Turing test: Include the corresponding image and linguistic criteria in human expert evaluation Based on these technical approaches, we demonstrated that the fine-tuned open-source MLLMs can be instrumented for 3D medical image interpretation. Recent work by Hamamci et al. showed that generative models are competent in 3D chest CT report generation at a state-of-the-art (SOTA) level (BLEU-1 = 46, BLEU-4 = 36.9, METEOR = 29.5, ROUGE-L = 45.9). 29-31 However, their customized transformer model required 7 days of training on a single NVIDIA A100 GPU, whereas our BrainGPT requires just 12 hours of fine-tuning on two NVIDIA A100 GPUs. Additionally, Google AI's Med-Gemini-3D can perform 3D CT report generation, but only 53% of its reports were considered clinically valid in human evaluations. 32 The high computational cost of using large-scale Google TPUv4 accelerator pods makes this approach infeasible for general research with limited resources. In contrast, our BrainGPT uses an end-to-end open-source Otter framework (CLIP ViT-L/14 vision encoder 32 and LLaMA-7B 33 ), which allows for experiment replication and checkpoint sharing. Moreover, the reduced training cost of BrainGPT enables efficient visual instruction tuning, enhancing model performance and tailoring responses to specialized or stylistic requirements (Extended Data Table 6). While we did not modify the Otter model structure, we attribute the SOTA-level performance to the combined effects of RVIT and CVIT. Singhal et al. first explored task-specific RVIT in the medical domain, showing that chatbot performance improved with medical QA in-context example primings. 27 Similarly, Med-PaLM M used image cues (CXR and pathology slides) alongside clinical instructions to guide MLLM on multimodal medical tasks. 16 Echoing these studies, our CVIT models (BrainGPT-template, BrainGPT-keyword) outperformed RVIT models in brain CT captioning. This suggests that fine-grained, specialist-grade instruction design may optimize model outcomes for clinical captioning tasks. We also highlighted that traditional metrics are unfit for evaluating clinical captioning tasks. Medical image reports assist in differential diagnosis and thus are characterized by complex paraphrasing, high token counts (>100), and numerous negative descriptions, which conflict with common metric evaluation contexts. 23,34 Additionally, we observed an “Interpretation Spree” behavior, where BrainGPT provided off-target (but not hallucinated) diagnostic narrations from multi-object brain CT contexts (detailed in Supplementary Table 1). This behavior is detrimental because (1) off-targeting may exclude the primary disease focus (e.g., stroke or brain tumor), and (2) expanded narration may dilute traditional metrics, leading to invalid evaluations. To address low traditional metric values from different narration aims, we preprocessed radiology reports with ”Sentence Pairing” and “Negation Removal.” We then applied FORTE to decompose BrainGPT reports into keyword categories addressing effective radiology descriptor frequency. The application gap is attributed to surface-form similarity and lexicographic analytic restrictions. 34 For the first time, we performed Pearson correlation analysis on different evaluation metrics, finding that FORTE encapsulates the semantic dimension more effectively than traditional CE metrics. 24 Meanwhile, the FORTE category concept is customizable and transferable across various medical image captioning tasks by substituting categorical keyword bank contents. (Extended Data Fig. 2) Human expert evaluation in natural language processing experiments has been conducted under distinct experiment designs and served diverse study purposes. Therefore, the resulting perspectives were often inconsistent and non-comparable across individual contexts. To this end, related works had instrumented quantitative (completeness, correctness, conciseness) and qualitative (content, language, structure) measurements to dissect the riveting characters that discern synthesized clinical reports from human reports. 35-37 By adopting similar designs with objective linguistic criteria, we found that both reviewer success rate and answer alternation reasons (“Suspicious wording” and “Both not mentioning critical features”) were associated with the writing style (“Familiarity and voice” and "Specificity or vagueness of details") rather than the sentence-level writing quality and coherence. 28 The significance of medical report writing style was also underlined in an independent prompting study. 36 Interestingly, we observed that input case imbalance can influence caption writing style, potentially related to over-fitting noticed in general model training (see also Extended Data Fig. 3). This study has several limitations, which should be addressed in future works. First, this pilot brain CT study has no counterpart MLLM module to benchmark, and hence we cannot justify the efficacy at a SOTA level; however, we applied external validation to ensure the caption validity in the brain CT module. Second, BrainGPT is trained on degeneration-oriented data and thus it fails to caption the malignancy tumor and acute traumatic features in CQ500. This phenomenon reflects that the training material may prime the dexterity of the resultant module. 38 Hence, we suggest enrolling diverse disease etiologies with the aim of differential diagnosis to improve MLLM generalization on border brain CT features. Last, we conducted CVIT and invented clinical-oriented evaluations (sentence-pairing, negation removal, and FORTE), but we did not experiment on whether changing the model backbone could benefit brain CT captioning. Future research avenues could be comparing multi-model results and fine-tuning the vision encoder and the language model for CT. Declarations Code and data availability All code used for experiments in this study can be found in a GitHub repository (https://github.com/charlierabea/FORTE), which also contains a model weight link to our best instruction-tuned model (BrainGPT-keyword). This study used data from Taipei Veterans General Hospital (TPEVGH) for the training process and CQ500 from Qure.ai for external validation. 39 Data from TPEVGH cannot be released due to IRB regulations, but researchers can access CQ500 via the provided reference. Note that any further distribution of datasets is subject to the terms of use and data-sharing agreements stipulated by the original creators. Acknowledgements Funded by grants of the Taiwan National Science and Technology Council (NSTC 112-2321-B-A49-007, NSTC 1 11-2320-B-A49-028-MY3, NSTC 112-2124-M-038-001, and NSTC 112-2314-B-032-001) and Taipei Veterans G eneral Hospital (V112C-026 and 112VACS-007) This study is based in part on data from the Big Data Center, Taipei Veterans General Hospital (BDC, TPEVGH). The interpretations and conclusions contained herein do not represent the position of Taipei Veterans General Hospital. Author Contributions C.Y.L collected data, designed visual instruction tuning instructions, ran experiments, analyzed results, created figures and wrote the manuscript. K.J.C proposed the FORTE framework, analyzed results, and wrote the manuscript. C.F.Y assisted in model fine-tuning and designed sentence pairing. H.Y.W conducted qualitative analysis and created figures. W.T.C assisted in model fine-tuning and provided technical advice. H.B. assisted in model survey and academic writing. L.C, Y.P.Y, Y.C.C, and S.P.C advised on study design and provided meaningful feedback. J.F.L directed the Turing Test and provided feedback on instruction design and the FORTE framework as a radiologist. K.W.C and S.H.C guided the project and provided critical suggestions for overall direction. All authors contributed to the interpretation and provided critical feedback on the analyses and manuscript. Competing Interests The authors declare no competing interests. References Cao, K. , et al. Large-scale pancreatic cancer detection via non-contrast CT and deep learning. Nat Med 29 , 3033-3043 (2023). Groh, M. , et al. Deep learning-aided decision support for diagnosis of skin disease across skin tones. Nat Med 30 , 573-583 (2024). Tian, F. , et al. Prediction of tumor origin in cancers of unknown primary origin with cytology-based deep learning. Nature Medicine (2024). Dai, L. , et al. A deep learning system for predicting time to progression of diabetic retinopathy. Nat Med 30 , 584-594 (2024). Rajpurkar, P. & Lungren, M.P. The Current and Future State of AI Interpretation of Medical Images. N Engl J Med 388 , 1981-1990 (2023). Boiselle, P.M. Computed tomography screening for lung cancer. JAMA 309 , 1163-1170 (2013). Wysoki, M.G. , et al. Head trauma: CT scan interpretation by radiology residents versus staff radiologists. Radiology 208 , 125-128 (1998). Boag, W. , et al. Baselines for chest x-ray report generation. in Machine learning for health workshop 126-140 (PMLR, 2020). Chen, Z., Song, Y., Chang, T.-H. & Wan, X. Generating radiology reports via memory-driven transformer. arXiv preprint arXiv:2010.16056 (2020). Selivanov, A. , et al. Medical image captioning via generative pretrained transformers. Scientific Reports 13 , 4171 (2023). Yang, S. , et al. Radiology report generation with a learned knowledge base and multi-modal alignment. Medical Image Analysis 86 , 102798 (2023). Johnson, A.E.W. , et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci Data 6 , 317 (2019). Demner-Fushman, D. , et al. Preparing a collection of radiology examinations for distribution and retrieval. J Am Med Inform Assoc 23 , 304-310 (2016). Chen, W., Shen, L., Li, X. & Yuan, Y. Fine-Grained Image-Text Alignment in Medical Imaging Enables Cyclic Image-Report Generation. arXiv:2312.08078 (2023). Li, C. , et al. LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day. arXiv:2306.00890 (2023). Tu, T. , et al. Towards Generalist Biomedical AI. NEJM AI 1 , AIoa2300138 (2024). Haydel, M.J. , et al. Indications for computed tomography in patients with minor head injury. N Engl J Med 343 , 100-105 (2000). König, M. Brain perfusion CT in acute stroke: current status. European Journal of Radiology 45 , S11-S22 (2003). Banerjee, S. & Lavie, A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization 65-72 (2005). Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. in Proceedings of the 40th annual meeting of the Association for Computational Linguistics 311-318 (2002). Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. in Text summarization branches out 74-81 (2004). Santos, G.O.d., Colombini, E.L. & Avila, S. CIDEr-R: Robust consensus-based image description evaluation. Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021) (2021). Liu, G. , et al. Clinically Accurate Chest X-Ray Report Generation. arXiv:1904.02633 (2019). Irvin, J. , et al. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. Proceedings of the AAAI Conference on Artificial Intelligence 33 , 590-597 (2019). Liu, H., Li, C., Wu, Q. & Lee, Y.J. Visual Instruction Tuning. arXiv:2304.08485 (2023). Li, B. , et al. Otter: A Multi-Modal Model with In-Context Instruction Tuning. arXiv:2305.03726 (2023). Singhal, K. , et al. Large language models encode clinical knowledge. Nature 620 , 172-180 (2023). Casal, J.E. & Kessler, M. Can linguists distinguish between ChatGPT/AI and human writing?: A study of research ethics and academic publishing. Research Methods in Applied Linguistics 2 , 100068 (2023). Ethem Hamamci, I., Er, S. & Menze, B. CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging. arXiv:2403.06801 (2024). Miura, Y., Zhang, Y., Tsai, E.B., Langlotz, C.P. & Jurafsky, D. Improving factual completeness and consistency of image-to-text radiology report generation. arXiv preprint arXiv:2010.10042 (2020). Nicolson, A., Dowling, J. & Koopman, B. Improving chest X-ray report generation by leveraging warm starting. Artificial Intelligence in Medicine 144 , 102633 (2023). Yang, L. , et al. Advancing Multimodal Medical Capabilities of Gemini. arXiv:2405.03162 (2024). Touvron, H. , et al. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 (2023). Abbasian, M. , et al. Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI. npj Digital Medicine 7 , 82 (2024). Van Veen, D. , et al. Adapted large language models can outperform medical experts in clinical text summarization. Nature Medicine (2024). Yan, B. , et al. Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting. arXiv:2310.17811 (2023). Boag, W., Kané, H., Rawat, S., Wei, J. & Goehler, A. A Pilot Study in Surveying Clinical Judgments to Evaluate Radiology Report Generation. in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency 458–465 (Association for Computing Machinery, Virtual Event, Canada, 2021). Youssef, A. , et al. External validation of AI models in health should be replaced with recurring local validation. Nature Medicine 29 , 2686-2687 (2023). Chilamkurthy, S. , et al. Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. The Lancet 392 , 2388-2396 (2018). Lin, T.-Y. , et al. Microsoft COCO: Common Objects in Context. in Computer Vision – ECCV 2014 (eds. Fleet, D., Pajdla, T., Schiele, B. & Tuytelaars, T.) 740-755 (Springer International Publishing, Cham, 2014). Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q. & Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019). Method Study Design and Oversight In this study, we trained BrainGPT to generate 3D brain CT reports. Then, we examined the caption efficacy by (1) adapting traditional evaluation metrics (2) proposing a clinical-oriented FORTE metric, (3) applying an external validation CQ500, and (4) conducting the linguistic-embedded Turing test. Study Patient We collected 18,885 brain CT scans (742,501 slices) from 9,689 patients with Alzheimer's Disease (mean age = 82.59 years [standard deviation = 9.3 years]; 56.4% male) at Taipei Veterans General Hospital in Taipei, Taiwan, between January 1, 2010, and December 31, 2022. All data were collected under institutional review board approval (2023-10-002 BC). Informed consent was exempted due to the retrospective nature of the data collection and the use of deidentified CT images. The CT images included a variety of common neurology conditions affecting the skull, brain parenchyma, nasal sinuses, and the eye, and were collected by radiologists who routinely obtain CT images and write image reports based on the images and the patient's medical records. Since Alzheimer's Disease is a progressive degenerative condition predominantly seen in the elderly, the dataset includes images of normal brains, past infarcts that still show manifestations, chronic brain conditions, and acute brain lesions. Clinical Visual Instruction Tuning (CVIT) To address the domain constraints of standard MLLM, we conducted multiple end-to-end visual instruction tuning processes on the multi-image mode Otter foundation model, enhancing its applicability to brain CT radiology features. 25,26 Based on the Flamingo model, the Otter paradigm connects the LLaMA-7B language encoder and the frozen CLIP ViT-L/14 vision encoder via a trainable perceiver resampler module and multiple cross-attention layers inserted into the LLaMA-7B architecture. Within the original LLaMA-7B structure, all modules except for the input/output embeddings were frozen to reduce training costs. The training duration for each resulting model was 12 hours on two NVIDIA A100 GPUs, achieving 3 epochs. To facilitate multi-image in-context learning capacity, we formulated the data into image-instruction-answer triplets, with the instruction tokenized and the image enhanced prior to input into the model. We designed four distinct fine-tuning conditions including regular visual instruction tuning (RVIT, Plain Instruction, and In-context example Instruction) and clinical visual instruction tuning (CVIT, Template Instruction, and Keyword Instruction), each corresponding to an adherence hierarchy to clinical essence. We named the final instruction-tuned models BrainGPT-plain, BrainGPT-example, BrainGPT-template, and BrainGPT-keyword. For each visual instruction tuning process, BrainGPT-plain was fine-tuned using plain instruction, conveying the model’s role as a radiology assistant; BrainGPT-example was fine-tuned using in-context example instruction, adopting a 3-shot example approach due to available RAM constraints as based on the work of Singhal et al.; 27 BrainGPT-template was fine-tuned using template instruction, involving a structured and predefined set of questions or points that need to be addressed; BrainGPT-keyword was fine-tuned using keyword instruction, focusing on essential areas or categorical guidelines that direct the model’s response generation process. Detailed instruction examples can be referenced in Extended Data Fig. 4. Dataset preparation Training Dataset. Since the Otter architecture requires image-text pair instances to be of the same size (24 slices), we sampled 365,928 slices from 15,238 scans representing 7,747 patients from a total of 597,335 slices from 15,247 scans representing 7,751 patients for the training process. The system was then tested on 87,312 slices sampled from 145,166 slices of 3,638 CT scans representing 1,938 patients. External Validation Dataset. The CQ500 dataset, consisting of 1,154 available CT scans from 491 patients, was downloaded from the Qure.ai website. 39 The dataset focuses on image features such as brain parenchyma (plain scans), bone (bone scans), and blood vessels (post-contrast scans). Only non-contrast CT scans with slice numbers between 23 and 40 were selected to build the external validation dataset (n=133). This ensures that slice thickness and details are similar to our training dataset and fit in the Otter framework. Ground truth was based on a read.csv file from the CQ500 dataset, and the majority rule was applied among the three raters to summarize the “Mass effect,” “Hemorrhage event,” and “Midline shift” labels. Feature-Oriented Radiology Task Evaluation (FORTE) We proposed the Feature-Oriented Radiology Task Evaluation (FORTE) framework to capture radiology keywords in terms of Degree (size and intensity), Landmark (location and anatomy), Feature (disease traits and primary findings), and Impression (final diagnosis) of the disease. (For details on the keyword list, see Extended Data Table 7). This list is not just a compilation of keywords but also includes synonyms, allowing the system to recognize a broader array of related terms that may appear in different contexts, addressing the challenge of lexical variability inherent in clinical reports. The F1 score is calculated for each category, providing a multi-faceted evaluation of the system's performance. Additionally, we compute Pearson’s correlation coefficient (Pearson’s r) for each FORTE category with traditional metrics, offering a deeper understanding of their applicability and limitations in radiological report evaluation. Traditional metrics We compared the clinical evaluation fitness of FORTE against the standard similarity-based evaluation metrics, including BLEU (Bilingual Evaluation Understudy, set range 0-100), METEOR (Metric for Evaluation of Translation with Explicit ORdering, set range 0-100), ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation with Longest Common Subsequence, set range 0-100), and CIDEr-R (Robust Consensus-based Image Description Evaluation, set range 0-1000) (Extended Data Table 8) 19-22 All standard evaluations were executed using the Microsoft Common Objects in Context (MS-COCO) toolkit. 40 Additionally, to address the list-by-list structure, diverse paraphrasing, and differential diagnosis-oriented negation description of brain CT reports, we incorporated sentence pairing (inspired by cosine similarity calculation in BERTScore) 41 and negation removal before applying the evaluation formulas. By these means, sentence pairing releases the sequential constraints of disease descriptors, and negation removal reduces the false positives in evaluation reports. Specifically, the reports were embedded and vectorized using the all-mpnet-base-v2 model from the SentenceTransformer library before pairing and scoring. Linguistic-embedded Turing test To examine whether the BrainGPT CT report recapitulates the linguistic texture of radiologist reports, we conducted a Turing test involving physicians and radiologists. Each participant was asked to distinguish BrainGPT reports from radiologist reports. The study was structured around four measuring axes: (1) Turing test: Can physicians tell the difference between BrainGPT reports and radiologist reports? (2) Confidence rate: How confident are the reviewers in their ratings? (3) Inter-leaved dependency: Do physicians alter their assessments and confidence rates after reviewing the original CT scans? (4) Linguistic criteria: What is the linguistic rationale behind physicians' impressions? 28 To explore the aforementioned questions, we collected survey and semi-structured rationale interview data. The physician survey was composed of six caption pairs, each comprising a BrainGPT report and a radiologist report. These examples included diverse disease instances including lacunar infarct, subdural hemorrhage, brain atrophy, and midline (cerebral falx) shift, thereby encompassing a range of both acute and chronic cerebral alterations for expert evaluation. (Details can be referred to Extended Data Fig. 5) Supplementary Table Supplementary Table 1 is not available with this version Additional Declarations There is NO Competing Interest. Supplementary Files ExtendedData.docx Cite Share Download PDF Status: Published Journal Publication published 06 Mar, 2025 Read the published version in Nature Communications → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4558754","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":313044258,"identity":"07cb6c3a-99bb-472a-95a0-faa9ba8d2c6f","order_by":0,"name":"Cheng-Yi Li","email":"","orcid":"https://orcid.org/0009-0000-9913-3695","institution":"Department of Computer Science, University of California, Los Angeles; School of Medicine, College of Medicine, National Yang Ming Chiao Tung University; Department of Medical Research, Taipei Veterans General Hospital","correspondingAuthor":false,"prefix":"","firstName":"Cheng-Yi","middleName":"","lastName":"Li","suffix":""},{"id":313044259,"identity":"ea61bb71-d1ff-47fa-8222-5e9c9aac2c33","order_by":1,"name":"Kao-Jung Chang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABFklEQVRIiWNgGAWjYDACZgY2IGnBYADiJBhIyPGDGQUEtUhAtDwosDGWbADrxWsPQgvjgw9piRsOgATxaNFtZ3724EeFhLw5+9nDLxIMDiduPr868cMDAwZ5frEDWLWYHWYzN+w5I2G4sycvzQKoxXjbjbebJYAOM5w5OwGHFh42Cd42oJoDOWYGQC2y226c3QDSkmBwG7cWyb//gGrOvwFrYdw84+zmH4S0SPM2ALXcyDF+kGCQpriBv3cbAVvYzKRljkkYbrjxxgyozMZY4gbvNqCnJHD75fzhZ5JvamzkDc7nGH/88QcYlf1nN9/8UWEjzy+NXQsyYJMAUxJglRIElYMA8wcwxX+AKNWjYBSMglEwcgAA5WpiYWIcAp8AAAAASUVORK5CYII=","orcid":"","institution":"Department of Medical Research, Taipei Veterans General Hospital; Institute of Clinical Medicine, National Yang Ming Chiao Tung University; Big Data Center, Taipei Veterans General Hospital; Department of Ophthalmology, Taipei Veterans General Hospital","correspondingAuthor":true,"prefix":"","firstName":"Kao-Jung","middleName":"","lastName":"Chang","suffix":""},{"id":313044260,"identity":"173e2ffe-9a82-4cb2-89a5-dc0eb45d2eb2","order_by":2,"name":"Cheng-Fu Yang","email":"","orcid":"","institution":"Department of Computer Science, University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Cheng-Fu","middleName":"","lastName":"Yang","suffix":""},{"id":313044261,"identity":"765243fd-ea9c-4eef-9144-e84affa5aaef","order_by":3,"name":"Hsin-Yu Wu","email":"","orcid":"","institution":"School of Medicine, College of Medicine, National Yang Ming Chiao Tung University; Department of Medical Research, Taipei Veterans General Hospital","correspondingAuthor":false,"prefix":"","firstName":"Hsin-Yu","middleName":"","lastName":"Wu","suffix":""},{"id":313044262,"identity":"d0f01632-b573-4a89-9776-782fbb099413","order_by":4,"name":"Wenting Chen","email":"","orcid":"","institution":"Department of Electrical Engineering, City University of Hong Kong","correspondingAuthor":false,"prefix":"","firstName":"Wenting","middleName":"","lastName":"Chen","suffix":""},{"id":313044263,"identity":"e61d5453-aaee-4ce2-974c-7eefd5ae9fba","order_by":5,"name":"Hritik Bansal","email":"","orcid":"","institution":"Department of Computer Science, University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Hritik","middleName":"","lastName":"Bansal","suffix":""},{"id":313044264,"identity":"0af9a6e3-6f79-4430-b95b-a70554979a48","order_by":6,"name":"Ling Chen","email":"","orcid":"","institution":"Institute of Hospital and Health Care Administration, National Yang Ming Chiao Tung University","correspondingAuthor":false,"prefix":"","firstName":"Ling","middleName":"","lastName":"Chen","suffix":""},{"id":313044265,"identity":"29b7c9f8-2e7c-4890-80aa-3c3a6bf46fa8","order_by":7,"name":"Yi-Ping Yang","email":"","orcid":"","institution":"Department of Medical Research, Taipei Veterans General Hospital","correspondingAuthor":false,"prefix":"","firstName":"Yi-Ping","middleName":"","lastName":"Yang","suffix":""},{"id":313044266,"identity":"166aafb0-37ed-450b-9d67-cf873054f05d","order_by":8,"name":"Yu-Chun Chen","email":"","orcid":"","institution":"School of Medicine, College of Medicine, National Yang Ming Chiao Tung University; Big Data Center, Taipei Veterans General Hospital; Institute of Hospital and Health Care Administration, National Yang Ming Chiao Tung University; Department of Family Medicine, Taipei Veterans General Hospital","correspondingAuthor":false,"prefix":"","firstName":"Yu-Chun","middleName":"","lastName":"Chen","suffix":""},{"id":313044267,"identity":"c9f46f79-02a0-4520-ae48-63243dfddec4","order_by":9,"name":"Shih-Pin Chen","email":"","orcid":"","institution":"Institute of Clinical Medicine, National Yang Ming Chiao Tung University; Department of Neurology, Neurological Institute, Taipei Veterans General Hospital","correspondingAuthor":false,"prefix":"","firstName":"Shih-Pin","middleName":"","lastName":"Chen","suffix":""},{"id":313044268,"identity":"7245a973-4b78-4d82-b16a-72349a3f1331","order_by":10,"name":"Jiing-Feng Lirng","email":"","orcid":"","institution":"Department of Radiology, School of Medicine, National Yang Ming Chiao Tung University; College of Medicine, National Yang Ming Chiao Tung University","correspondingAuthor":false,"prefix":"","firstName":"Jiing-Feng","middleName":"","lastName":"Lirng","suffix":""},{"id":313044269,"identity":"8b5e6b07-11dd-4a07-a08d-eadf8820dc69","order_by":11,"name":"Kai-Wei Chang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA4ElEQVRIiWNgGAWjYBACCQYGNhAtB+EWgIgE4rQYQ7gGJGhJbCBai+SM5GcPfu6oTe/vP8Am8cPAjoGfPccArxZpiTRzw94zx3Nn3Ehgk+wxSGaQ7HmDX4ucdA6bBG/bsdwNEgxs0gwGzAwGNwjYAtIi+bftWLoB/wGQlnoGe0JapIFapHnbahIMGBJAWg4zGEgQ0CI5/5mZtGzbAcMZNxKbLXsMjvNInHlWgFeLxJnDzyTfttXJ8/cfPnjjR0W1HH978ga8WqDgMBAzNoBYPMQoB4E6YhWOglEwCkbBSAQAZGI9Xx7Dy8wAAAAASUVORK5CYII=","orcid":"","institution":"Department of Computer Science, University of California, Los Angeles","correspondingAuthor":true,"prefix":"","firstName":"Kai-Wei","middleName":"","lastName":"Chang","suffix":""},{"id":313044257,"identity":"88e3f718-6cd5-4620-88d3-43202df36797","order_by":12,"name":"Shih-Hwa Chiou","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7klEQVRIiWNgGAWjYDACCQY2hgQGBjkGBh4StRgDtTA2QMXgDNxagCCxgWgt8rObnz14UHE4fcP5tccfF1Qw2PVLJLA/xudIgzvHzA0SzhzO3XDjXWLzjDMMyTNnJDA249UikcMmkdh2OHfmjDOGzbxtDMkGZw4wNufgc9gMiJZ0SbCWf0RoYbgB0ZLAz98D1NLAYGdwvAG/FoMbaWYSCWfSDfsleAxnzzgmkSDZ3tg4+w9ehyU/k/xRYS3Pxn/G4HNBjY09PzPzgY8z8DkMDiQSGJiBJDCCCMQkAvAfAGlhsCdS+SgYBaNgFIwgAAAnxU5ftm9opwAAAABJRU5ErkJggg==","orcid":"","institution":"Department of Medical Research, Taipei Veterans General Hospital; Institute of Clinical Medicine, National Yang Ming Chiao Tung University; Department of Ophthalmology, Taipei Veterans General Hospital","correspondingAuthor":true,"prefix":"","firstName":"Shih-Hwa","middleName":"","lastName":"Chiou","suffix":""}],"badges":[],"createdAt":"2024-06-10 14:50:49","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4558754/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4558754/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41467-025-57426-0","type":"published","date":"2025-03-06T05:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":59608640,"identity":"1e608df6-7784-4f8e-b211-27227d536685","added_by":"auto","created_at":"2024-07-03 19:13:08","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":927411,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eStudy Overview.\u003c/strong\u003e Given a 3D brain CT scan, BrainGPT can follow instructions to generate an image report for the volumetric data. We then evaluate the generated report in comparison with the ground truth by radiologists. Within the proposed feature-oriented radiology task evaluation (FORTE), reports are reassembled into four categories of clinical keywords: degree, landmark, feature, and impression. The system accommodates synonyms, allowing the recognition of a broader array of related terms that may appear in different contexts to address the challenge of lexical variability inherent in clinical reports. In this image-text example, image features and corresponding report descriptions are labeled with the same color.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-4558754/v1/7daf017ff46435a022b07e55.png"},{"id":59609269,"identity":"aed6981d-602b-4a78-b299-5ff43e643be0","added_by":"auto","created_at":"2024-07-03 19:29:08","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":955820,"visible":true,"origin":"","legend":"\u003cp\u003eClinical visual instruction tuning BrainGPT from Otter. Our approach adopts the end-to-end Otter paradigm to train the BrainGPT model. We designed four distinct fine-tuning conditions: two for regular visual instruction tuning (RVIT) – Plain Instruction and In-context Example Instruction, and two for clinical visual instruction tuning (CVIT) – Template Instruction and Keyword Instruction. To enable multi-image in-context learning, we formatted the data as image-instruction-answer triplets, where instructions were tokenized, and images were enhanced before being inputted into the model. The Otter framework integrates visual data (using a frozen CLIP ViT-L/14 encoder) and language data (using the LLaMA-7B large language model) through a trainable perceiver resampler module. In the LLaMA-7B architecture, cross-gated attention layers were added to focus on CT slices, while all other modules, except for input/output embeddings, were frozen to minimize training expenses. The fine-tuned models were evaluated using traditional metrics and a novel FORTE evaluation method, designed for this study.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-4558754/v1/2dcfce499b615c46547246a3.png"},{"id":59608638,"identity":"51faf759-cdf9-4fb4-8208-e17a6541116d","added_by":"auto","created_at":"2024-07-03 19:13:08","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":952251,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eImproving conventional captioning metrics. \u003c/strong\u003eWe performed a traditional evaluation comparing the BrainGPT report and the radiologist's report. (a) The generated report and the ground truth, each containing multiple descriptions, were restructured with sentence pairing to simulate a common image captioning scenario. (b) Traditional metric evaluation results show the training efficacy with statistically significant differences between the Otter model and the fine-tuned BrainGPT models (indicated by red annotations). The preprocessing involving sentence pairing highlighted the clinical value in the list-by-list differential diagnosis reports, as the CVIT (Template Instruction and Keyword Instructions) achieved significantly better results. (c) The dashed line denotes equivalence, and each data point corresponds to a text-image instance. The distribution of sentence pairing effects demonstrates the unique status of CIDEr, which is the only metric showing a moderate Pearson’s r. (* p \u0026lt; 0.05, ** p \u0026lt; 0.01, *** p \u0026lt; 0.001)\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-4558754/v1/9f9c4ab36b3ef8234811d5ae.png"},{"id":59608644,"identity":"f1f4fd66-92e2-43c5-9fdb-0f4a4a90a4f7","added_by":"auto","created_at":"2024-07-03 19:13:08","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":1401066,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eFeature-Oriented Radiology Task Evaluation (FORTE) \u003c/strong\u003e(a) Distribution of keyword frequency in the ground truth descriptions versus the BrainGPT model outputs. Most high-frequency keywords are grammatical filler words, but the importance of the keyword “no” is highlighted. (b) Results for FORTE, with the four categories designed to measure the clinical essence of generated reports. Negation removal was implemented to limit the description range to only positive findings. (c) Effects of applying negation removal on traditional evaluation metric results. (d) Pearson analysis between the traditional metrics and the FORTE keyword categories. (* p \u0026lt; 0.05, ** p \u0026lt; 0.01, *** p \u0026lt; 0.001)\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-4558754/v1/a1b7252fde05cc367d2ec932.png"},{"id":59608990,"identity":"fca3f2ca-6a0f-4c3d-ad60-7770073bddef","added_by":"auto","created_at":"2024-07-03 19:21:08","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":1583249,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eExamining Multi-object Multi-Slice Captioning Capability of BrainGPT on the Stroke-featured CQ500 External Validation dataset. \u003c/strong\u003e(a) Bar chart summarizing and comparing the feature composition between our 3D-brainCT (training and internal testing) dataset and the CQ500 (external validation) dataset. The feature captioning accuracy (mass effect, hemorrhage, and midline shift) of the four BrainGPT models was calculated and represented as a line plot. (b) Head-to-head comparison of BrainGPT model-generated image reports for distinct brain lesion features (lacunar infarction, subdural hemorrhage (SDH), subarachnoid hemorrhage (SAH), midline shift, and herniation). (c) BrainGPT demonstrated a strong captioning ability for volumetric brain CT slices, generating reports with lesion descriptions linked to individual image slices.\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-4558754/v1/df8ff99b3dceb611312ebab3.png"},{"id":59608989,"identity":"9a6036af-8795-4b48-993d-1b711d7a1c7e","added_by":"auto","created_at":"2024-07-03 19:21:08","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":221246,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003e\u003cstrong\u003eTuring\u003c/strong\u003e\u003c/em\u003e\u003cstrong\u003e test. \u003c/strong\u003e(a) the accuracy of responses and levels of confidence among physicians before and after review of the original CT scans. (b) the linguistic criteria employed by physicians and the corresponding accuracy change.\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-4558754/v1/8535eb4a46dc69e07f6ec781.png"},{"id":77955195,"identity":"aa29ee2b-412b-411a-8e4b-c2f6d616773c","added_by":"auto","created_at":"2025-03-07 08:08:01","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":7012425,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4558754/v1/8ab50b03-55b3-4ac6-8801-eaacd9b841b2.pdf"},{"id":59608641,"identity":"edcfd4db-2d42-4348-84da-f58a139bfef6","added_by":"auto","created_at":"2024-07-03 19:13:08","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":5125930,"visible":true,"origin":"","legend":"","description":"","filename":"ExtendedData.docx","url":"https://assets-eu.researchsquare.com/files/rs-4558754/v1/92fe524c6f18e92152e8ed3e.docx"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation","fulltext":[{"header":"Main","content":"\u003cp\u003eThe rapid ascent of artificial intelligence (AI) technologies has profoundly influenced the interpretation of multimodal information such as computed tomography (CT),\u003csup\u003e1\u003c/sup\u003e gross dermatologic conditions,\u003csup\u003e2\u003c/sup\u003e cytology slides,\u003csup\u003e3\u003c/sup\u003e ophthalmology images,\u003csup\u003e4\u003c/sup\u003e etc. While modern models excel in tasks like classification, regression, and segmentation, they fall short in providing the natural-language expressivity and interactivity required in hands-on clinical workflows.\u003csup\u003e5\u003c/sup\u003e This discrepancy is critical because instant scan interpretation is essential to clinical decision-making, with the rate decision step being the radiologist's report writing.\u003csup\u003e6,7\u003c/sup\u003e To address this need, early generative models.\u003csup\u003e8\u0026ndash;11\u003c/sup\u003e have been applied to chest X-ray (CXR) report generation in datasets such as the MIMIC-CXR and OpenI to relieve radiologists' time-consuming and tedious burden.\u003csup\u003e12,13\u003c/sup\u003e These studies demonstrated a potential interface where human-computer collaboration can form medical reports, which can be further refined with multi-modal large language models (MLLM).\u003csup\u003e14\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eTo examine the full adequacy of MLLM in radiology report generation, we conducted a thorough review of relative works. Despite the primary success in CXR report generation, we identified three crucial limitations in the existing literature, including (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e) data complexity, (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e) model capacity, and (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e) evaluation metric fidelity. These innate limitations have hindered the development of totipotent MLLMs.\u003c/p\u003e \u003cp\u003eWhile the human body is inherently three-dimensional (3D), existing 2D CXR image datasets\u003csup\u003e12,13\u003c/sup\u003e have yet to encompass (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e) pathologic features in complex neuro-vascular anatomy, i.e., the brain, heart, and eye, (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e) spatial landmarks within the 3D image context, and (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e) the intensity degree or size of lesions. Moreover, the single-slice image content from current CT image-text data is usually selected from 3D data to include the described feature. This may be subject to \u003cem\u003esharpshooter fallacy\u003c/em\u003e, where the selected slice often indicates the caption target of interest. Hence, it is imperative to curate a 3D image set to inspect the dexterities of MLLMs in real-world diagnostic contexts.\u003c/p\u003e \u003cp\u003eCrucially, MLLMs must be capable of multi-image reasoning or video understanding when used to generate reports for real-world 3D data. Recent medical MLLM studies such as LLaVA-Med (Microsoft) and Med-PaLM M (Google Research) have demonstrated promising X-ray and single-slice CT report generation performance.\u003csup\u003e15,16\u003c/sup\u003e However, during testing, the brain CT data is presented in a 3D format, which poses challenges for the MLLM to identify the specific slice containing the lesion of interest precisely \u003cem\u003ein situ\u003c/em\u003e.\u003c/p\u003e \u003cp\u003e3D CT scans of the brain, with reports structured in a list-by-list format focusing on differential diagnoses, serve as the first-line diagnostic for diverse intracranial conditions.\u003csup\u003e17,18\u003c/sup\u003e The brain lesion's degree, size, and location are paramount to forming precise diagnoses and guiding subsequent clinical decisions. Therefore, the value of key radiology descriptors outweighs grammatical filler words. In this regard, traditional evaluation metrics, designed to evaluate short translation\u003csup\u003e19,20\u003c/sup\u003e, abstract summary tasks\u003csup\u003e21\u003c/sup\u003e, and common image captioning\u003csup\u003e22\u003c/sup\u003e cannot gauge the clinical essence of Brain CT reports.\u003csup\u003e23\u003c/sup\u003e On the other hand, the CheXpert clinical efficacy evaluation is designed to capture pathology keyword retrieval rates to represent the CXR report quality.\u003csup\u003e11,24\u003c/sup\u003e However, 14-feature retrieval rates cannot reflect the multi-semantic context (inclusive or exclusive; progression or resolution; relative location and size) of disease features. Moreover, the uncategorized keyword bank concept may not be transferable from one image modality to other modalities. To alleviate such constraints, radiology semantic categories must be incorporated into a novel scoring system to measure the clinical essence of radiology reports.\u003c/p\u003e \u003cp\u003eIn this study, we aim to advance the application of MLLMs in radiology by addressing several critical areas:\u003c/p\u003e \u003cp\u003e(\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e) We curated a large-scale 3D-brainCT dataset (18,885 text-scan pairs) abundant with lesion degree, spatial landmarks, and diagnostic impressions of neuronal and vascular features to present a real-world challenge for MLLMs. (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e and Extended Data Table\u0026nbsp;1)\u003c/p\u003e \u003cp\u003e(\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e) We applied the clinical visual instruction tuning (CVIT) concept to train the BrainGPT model from the open-source Otter.\u003csup\u003e25,26\u003c/sup\u003e The Otter model\u0026rsquo;s multi-image captioning capacity and clinical instruction design enable the generation of clinically sensible captions from volumetric brain CT scans. (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e) The BrainGPT was externally validated on CQ500 (Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e) and an 11-physician rater Turing Test-like linguistic-embedded evaluation. (Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003e)\u003c/p\u003e \u003cp\u003e(\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e) We proposed feature-oriented radiology task evaluation (FORTE) to measure the clinical relevance of captions based on distinct disease objectives, including 4 generalizable keyword categories (degree, landmark, feature, and impression). (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e and Extended Data Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e) We also suggested that preprocessing processes, such as sentence pairing and negation removal, are beneficial in adapting traditional metrics to the detailed, itemized, and differential diagnosis-oriented format of brain CT reports. (Figs.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e and \u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003ec)\u003c/p\u003e \u003cp\u003eOverall, this study proposes a holistic framework that addresses the implementation details of applying MLLMs in brain CT and other complex anatomy images.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e"},{"header":"Results","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eTraining BrainGPT with Clinical Visual Instruction Tuning\u003c/h2\u003e \u003cp\u003eTo implement medical MLLM in 3D brain CT report generation, we collected a real-world 3D-BrainCT dataset and trained BrainGPT models by applying visual instruction tuning on the Otter foundation model (Extended Data Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e). Our fine-tuning conditions included regular visual instruction tuning (RVIT): (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e) Plain Instruction (conveys the model\u0026rsquo;s role as a radiology assistant) and (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e) In-context example Instruction\u003csup\u003e27\u003c/sup\u003e (3-shot examples added to the Plain Instruction), or clinical visual instruction tuning (CVIT): (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e) Template Instruction (structured clinical-defined QA templates added to the Plain Instruction) and (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e) Keyword Instruction (categorical guidelines focused on keywords added to the Plain Instruction). As a result, we derived four BrainGPT models (-plain, -example, -template, -keyword) that are tuned to perform CT interpretation with different prior levels of clinical sensibility (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003csup\u003e25\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eAt the whole report level, all of the four fine-tuned BrainGPT models outperformed the baseline Otter model in traditional metrics (Mann-Whitney U test \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.01, see Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eb). In particular, the baseline Otter model scored the lowest in BLEU-4 (0) and CIDEr-R (5.9) metrics, indicating that the baseline model-generated CT report had low n-gram overlap and underfit semantic term usage frequency. In addition, we observed a mismatch between the conventional metric and the report quality by reviewing generated reports (see Extended Data Table\u0026nbsp;1). Moreover, no traditional metrics, except CIDEr-R, rated the BrainGPT performance in accord with the crescendo elaborateness of RVIT and CVIT (Mann-Whitney U test p\u0026thinsp;\u0026gt;\u0026thinsp;0.05 across four visual instruction tuning conditions, summarized in Extended Data Table\u0026nbsp;2). Therefore, we demonstrated that traditional metrics were inherently insensitive to the clinical essence of the generated radiology reports.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eSentence pairing\u003c/h2\u003e \u003cp\u003eTo investigate whether the list-by-list report architecture of brain CT images accounted for low traditional metric scores, we applied sentence pairing to decompose the multi-sentence paragraph into smaller semantic granularity. (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003ea) As a result, the process of sentence pairing increased traditional metric scores by an average of 5.28 points in METEOR, 6.48 points in ROUGE-L, and an astonishing 114 points in CIDEr-R (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eb and Extended Data Table\u0026nbsp;2). The traditional metric scores for every BrainGPT model were significantly improved across most traditional metrics (\u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.001, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003ec and Extended Data Table\u0026nbsp;3). In addition, the score distribution indicated significant linear correlation (Pearson correlation coefficient r\u0026thinsp;\u0026gt;\u0026thinsp;0.7) in BLEU-2, BLEU-3, BLEU-4, METEOR, and ROUGE-L with statistical significance (\u003cem\u003ep\u003c/em\u003e-value\u0026thinsp;\u0026lt;\u0026thinsp;0.05). However, apart from the other n-gram-based traditional metric, the CIDEr-R score gained a more positive shift from the process of sentence pairing (r\u0026thinsp;=\u0026thinsp;0.577).\u003c/p\u003e \u003cp\u003eMoreover, we observed a positive correlation between advanced CVIT and increased traditional metric scores. In particular, the CIDEr-R score depicted the most prominent ascending trend across the hierarchically fine-tuned BrainGPT-plain (125.86), BrainGPT-example (132.38), BrainGPT-template (147.92), and BrainGPT-keyword (153.3). This sentence pairing procedure not only relieved the sequential constraint between the report input and generated output but also accentuated the clinical value in the list-by-list differential diagnosis reports.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003eFeature-Oriented Radiology Task Evaluation (FORTE)\u003c/h2\u003e \u003cp\u003eGiven that CIDEr-R captures the hierarchical clinical essence across visual instruction tuning conditions, we hypothesized that its frequency-inverse document frequency (TF-IDF) component reacts to the usage of rare radiology keywords in reports. Therefore, we analyzed the term frequency in both the ground truth and test outputs (Fig.\u0026nbsp;4a\u003csub\u003e1\u003c/sub\u003e). We found that the baseline Otter model had a low recall rate for keywords, whereas the BrainGPT models, after visual instruction tuning, demonstrated significant radiology keyword usage in image captioning. Notably, the BrainGPT-keyword (Fig.\u0026nbsp;4a\u003csub\u003e2\u003c/sub\u003e: y\u0026thinsp;=\u0026thinsp;0.95x-1.18; r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;0.85) maintained a high recall rate across varying frequencies of radiology keywords.\u003c/p\u003e \u003cp\u003eDue to the potential instability caused by keyword frequency variance in report evaluations, we introduced the feature-oriented radiology task evaluation (FORTE). This method gauges the medical content of the generated report by focusing on the density of radiology information. We categorized radiology keywords into degree, landmark, feature, and impression subsets to offer a multi-faceted evaluation of the system's performance.\u003c/p\u003e \u003cp\u003eBy evaluating the F1 score of each FORTE-measured keyword category (see Extended Data Table\u0026nbsp;4), we noted that the advanced CVIT models, BrainGPT-template and BrainGPT-keyword, scored higher average F1 score than the RVIT models. (Mann-Whitney U test \u003cem\u003ep\u003c/em\u003e-value\u0026thinsp;\u0026lt;\u0026thinsp;0.001, as detailed in Extended Data Table\u0026nbsp;5) Specifically, BrainGPT-keyword captured the most radiology details, with FORTE F1 scores of: degree\u0026thinsp;=\u0026thinsp;0.548, landmark\u0026thinsp;=\u0026thinsp;0.533, feature\u0026thinsp;=\u0026thinsp;0.574, and impression\u0026thinsp;=\u0026thinsp;0.649. These collective results indicate that clinical instruction-tuned BrainGPT models generate brain CT reports with a higher level of radiology term usage and better alignment with original diagnostic reports.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003eCompare FORTE against traditional evaluation metrics\u003c/h2\u003e \u003cp\u003eTo determine if FORTE reveals report qualities uncovered by traditional metrics, we conducted a Pearson correlation coefficient analysis to compare FORTE with traditional metrics (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003ed). We observed that traditional metrics had high intra-correlations (r\u0026thinsp;\u0026gt;\u0026thinsp;0.7, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.001), but lower correlation with FORTE (r\u0026thinsp;\u0026lt;\u0026thinsp;0.5, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.001) with the FORTE. In addition, the four FORTE domains (degree, landmark, feature, and impression) had lower intra-correlations (r\u0026thinsp;\u0026lt;\u0026thinsp;0.5, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.001 compared to traditional metrics). This comparison indicated that the FORTE addresses broader and distinct disease aspects than those traditional metrics depicted.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003eNegation Removal: Effect of \u0026ldquo;Interpretation Spree\u0026rdquo;\u003c/h2\u003e \u003cp\u003eUpon further review of keyword usage frequency, we found that \u0026ldquo;no\u0026rdquo; was notably among the most frequently used words, along with \u0026ldquo;of,\u0026rdquo; \u0026ldquo;and,\u0026rdquo; \u0026ldquo;in,\u0026rdquo; and \u0026ldquo;the.\u0026rdquo; This contradicts the common belief of \u0026ldquo;reporting bias,\u0026rdquo; where it is assumed that language models, similar to laypersons, are less likely to mention negative features. Clinically, radiologists often focus on context-oriented descriptions to \u0026ldquo;rule out\u0026rdquo; diagnostic targets, a practice not mirrored by BrainGPT. Consequently, BrainGPT\u0026rsquo;s reports were overlaid with excessive negative descriptions rather than the concise, context-specific language used by radiologists, a phenomenon we termed the \u0026ldquo;Interpretation spree.\u0026rdquo;\u003c/p\u003e \u003cp\u003eTo address this, we applied negation removal to limit descriptions to positive findings. (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003ec) The average score increases were: BLEU-1\u0026thinsp;=\u0026thinsp;29.25%, BLEU-2\u0026thinsp;=\u0026thinsp;34.9%, BLEU3\u0026thinsp;=\u0026thinsp;50.54%, BLEU-4\u0026thinsp;=\u0026thinsp;57.26%, METEOR\u0026thinsp;=\u0026thinsp;24.97%, ROUGE_L\u0026thinsp;=\u0026thinsp;29.04%, CIDEr-R\u0026thinsp;=\u0026thinsp;46.6%, with statistical significance (See Extended Data Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eAdditionally, negation removal improved the BrainGPT FORTE keyword average F1 score by 0.118 for degree, 0.194 for landmark, 0.139 for feature, and 0.163 for impression (see Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003eb and Extended Data Table\u0026nbsp;4). Negation removal led to an overall average F1 score gain of 0.153, with the best instruction-tuned model, BrainGPT-keyword, seeing its F1 score rise from 0.576 to 0.71. Together, we demonstrated that negation removal allowed evaluation metrics to focus on positive findings. Meanwhile, negation removal also enhanced average traditional scores and FORTE F1 scores.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eAssess BrainGPT Generalization by CQ500 External Validation\u003c/h2\u003e \u003cp\u003eWe conducted zero-shot external validation on the CQ500 brain CT dataset (n\u0026thinsp;=\u0026thinsp;133) to assess BrainGPT's ability to perform report captioning in generalized clinical conditions. We estimated the clinical keyword retrieval rate by comparing BrainGPT's reports to the ground truth labels of the CQ500 CT scans (Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003ea). Notably, BrainGPT mentioned keywords (e.g., ventricular enlargement, atrophy, infarction, and mass effect) in the generated reports for the CQ500 validation dataset with frequencies similar to the training dataset (3D-brainCT). These zero-shot captioning results indicated that BrainGPT had inscribed the CT report phraseology and writing structure for brain disease differential diagnosis.\u003c/p\u003e \u003cp\u003eCQ500 is an intracranial hemorrhage (ICH) dataset encompassing \u0026ldquo;mass effect,\u0026rdquo; \u0026ldquo;hemorrhage,\u0026rdquo; and \u0026ldquo;midline shift\u0026rdquo; features, which were less indexed in our geriatric population 3D-brainCT dataset. Nevertheless, the four instruction-tuned BrainGPT models achieved fair accuracy in reporting mass effect (0.71\u0026ndash;0.75), midline shift (0.35\u0026ndash;0.38), and hemorrhagic events (0.43\u0026ndash;0.5). Furthermore, consistent with our previous findings, negation removal (aligning with differential diagnosis interest) significantly increased report accuracy for midline shift (0.86\u0026ndash;0.91) and hemorrhagic events (0.59\u0026ndash;0.61). Notably, in terms of diagnosing \u0026ldquo;mass effect,\u0026rdquo; \u0026ldquo;hemorrhage,\u0026rdquo; and \u0026ldquo;midline shift,\u0026rdquo; the advanced CVIT models (BrainGPT-template and BrainGPT-keyword) generally outperformed the RVIT fine-tuned models (BrainGPT-plain and BrainGPT-example). This suggests that CVIT instructions acted as a pre-model rectifier, enhancing the focus on valuable clinical etiologies.\u003c/p\u003e \u003cp\u003eTo further investigate diagnostic credibility, we examined report results for lacunar infarct, subdural hemorrhage (SDH), subarachnoid hemorrhage (SAH), and midline shift with mass effect (Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003eb). Interestingly, in the lacunar infarction case, BrainGPT-example accurately identified the hypodense old lacunar infarct in the putamen and thalamus regions. However, \u0026ldquo;putamen\u0026rdquo; was misspelled as \u0026ldquo;putmen,\u0026rdquo; likely due to a typo learned from the original training report. The model also generated successful captions for brain lesions, including SDH at the right occipital, SAH, and left uncal herniation. To examine if BrainGPT was subject to the \u003cem\u003esharpshooter fallacy\u003c/em\u003e in 3D CT scans, we isolated multi-object cases with different lesion features in various CT cuts (Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003ec). We noted that in some multi-lesional brain CT, the BrainGPT reports could recognize distinct hemorrhage subtypes (contusion, intraventricular and subarachnoid hemorrhage) with a sparse comprehension of the lesion location (frontotemporal lobe). In another case, BrainGPT described a regressing chronic ICH and a periventricular angiopathic leukoencephalopathy with white matter hypo-density. These multi-object, multi-slice caption reports demonstrated BrainGPT's potential to provide differential diagnoses for hemorrhages and hypodense image features.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eLinguistic-embedded\u003c/b\u003e \u003cb\u003eTuring\u003c/b\u003e \u003cb\u003etest\u003c/b\u003e\u003c/p\u003e \u003cp\u003eOur results demonstrated that the BrainGPT models were capable of emulating the writing style of radiologists, using domain-specific language, and providing preliminary diagnostic information. To further evaluate whether these BrainGPT-generated reports could be distinguished by healthcare providers, we conducted a \u003cem\u003eTuring\u003c/em\u003e test. (For the questionnaire design, see Extended Data Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e)\u003c/p\u003e \u003cp\u003eIn this test, we enrolled 11 physicians (2 radiologists, 2 neurologists, and 7 other licensed medical doctors) (Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003ea). These doctors were asked to determine whether 6 provided report cases (two lacunar infarcts, cortex atrophy, subdural hemorrhage, and midline shift) were written by machines or humans. Interestingly, out of 66 evaluations, 74.24% of BrainGPT-generated captions were mistakenly identified as human-written, whereas only 46.97% of human-written reports were correctly identified. When we provided the evaluators with the input CT images, nearly 18% of their decisions were changed for BrainGPT reports, resulting in 56% of machine-written reports being indistinguishable from human-written reports. This input image-directed decision change was not observed in real human-written reports, which had a 50% correct identification rate. Overall, integrating the scan-text context into the \u003cem\u003eTuring\u003c/em\u003e test led to more confident evaluations on the 5-point Likert scale (\u003cem\u003ep\u003c/em\u003e-value). This suggests that reviewing actual image contexts helps human evaluators detect subtle differences between machine and human-written reports, likely due to off-target descriptions and inadequate precision in machine-generated captions.\u003c/p\u003e \u003cp\u003eWe also analyzed the qualitative traits influencing evaluator decisions (Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003eb). We found that the traits of \u0026ldquo;Familiarity and voice\u0026rdquo; (29%), \"Specificity or vagueness of details\" (37.60%), \u0026ldquo;Continuity and Coherence\u0026rdquo; (33.33%), and \u0026ldquo;Writing quality at sentence level\u0026rdquo; (33.33%) were significant factors in the evaluators' decisions.\u003csup\u003e28\u003c/sup\u003e Notably, when reviewers considered \"Familiarity and voice,\" the average success rate increased by 73.7% (from 22.22\u0026ndash;38.6%). Similarly, considering \"Specificity or vagueness of details\" increased the average success rate by 47% (from 26.67\u0026ndash;39.22%). These linguistic criteria underscore the importance of fine-tuning the radiologist writing style and incorporating radiology-specific details in medical report generation.\u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eIn this study, we assembled a holistic framework to address various unmet needs in achieving robust report generation for 3D brain CT. Based on our collective experiences, we recommend applying the following sequential protocol (or an equivalent process) to ensure caption and evaluation rigor:\u003c/p\u003e\n\u003col\u003e\n \u003cli\u003eCVIT: Incorporate clinical domain knowledge in implementing visual instruction tuning\u003c/li\u003e\n \u003cli\u003eCombine “sentence pairing” and “negation removal”: Adapt traditional metrics to radiology reports\u003c/li\u003e\n \u003cli\u003eEmploy FORTE: Gauge the clinical essence of caption results in multiple aspects\u003c/li\u003e\n \u003cli\u003eConduct linguistic-embedded \u003cem\u003eTuring\u003c/em\u003e test: Include the corresponding image and linguistic criteria in human expert evaluation\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eBased on these technical approaches, we demonstrated that the fine-tuned open-source MLLMs can be instrumented for 3D medical image interpretation.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eRecent work by Hamamci et al. showed that generative models are competent in 3D chest CT report generation at a state-of-the-art (SOTA) level (BLEU-1 = 46, BLEU-4 = 36.9, METEOR = 29.5, ROUGE-L = 45.9).\u003csup\u003e29-31\u003c/sup\u003e However, their customized transformer model required 7 days of training on a single NVIDIA A100 GPU, whereas our BrainGPT requires just 12 hours of fine-tuning on two NVIDIA A100 GPUs. Additionally, Google AI's Med-Gemini-3D can perform 3D CT report generation, but only 53% of its reports were considered clinically valid in human evaluations.\u003csup\u003e32\u003c/sup\u003e The high computational cost of using large-scale Google TPUv4 accelerator pods makes this approach infeasible for general research with limited resources. In contrast, our BrainGPT uses an end-to-end open-source Otter framework (CLIP ViT-L/14 vision encoder\u003csup\u003e32\u003c/sup\u003e and LLaMA-7B\u003csup\u003e33\u003c/sup\u003e), which allows for experiment replication and checkpoint sharing. Moreover, the reduced training cost of BrainGPT enables efficient visual instruction tuning, enhancing model performance and tailoring responses to specialized or stylistic requirements (Extended Data Table 6).\u003c/p\u003e\n\u003cp\u003eWhile we did not modify the Otter model structure, we attribute the SOTA-level performance to the combined effects of RVIT and CVIT. Singhal et al. first explored task-specific RVIT in the medical domain, showing that chatbot performance improved with medical QA in-context example primings.\u003csup\u003e27\u003c/sup\u003e Similarly, Med-PaLM M used image cues (CXR and pathology slides) alongside clinical instructions to guide MLLM on multimodal medical tasks.\u003csup\u003e16\u003c/sup\u003e Echoing these studies, our CVIT models (BrainGPT-template, BrainGPT-keyword) outperformed RVIT models in brain CT captioning. This suggests that fine-grained, specialist-grade instruction design may optimize model outcomes for clinical captioning tasks.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWe also highlighted that traditional metrics are unfit for evaluating clinical captioning tasks. Medical image reports assist in differential diagnosis and thus are characterized by complex paraphrasing, high token counts (\u0026gt;100), and numerous negative descriptions, which conflict with common metric evaluation contexts.\u003csup\u003e23,34\u003c/sup\u003e Additionally, we observed an “Interpretation Spree” behavior, where BrainGPT provided off-target (but not hallucinated) diagnostic narrations from multi-object brain CT contexts (detailed in Supplementary Table 1). This behavior is detrimental because (1) off-targeting may exclude the primary disease focus (e.g., stroke or brain tumor), and (2) expanded narration may dilute traditional metrics, leading to invalid evaluations.\u003c/p\u003e\n\u003cp\u003eTo address low traditional metric values from different narration aims, we preprocessed radiology reports with ”Sentence Pairing” and “Negation Removal.” We then applied FORTE to decompose BrainGPT reports into keyword categories addressing effective radiology descriptor frequency. The application gap is attributed to surface-form similarity and lexicographic analytic restrictions.\u003csup\u003e34\u003c/sup\u003e For the first time, we performed Pearson correlation analysis on different evaluation metrics, finding that FORTE encapsulates the semantic dimension more effectively than traditional CE metrics.\u003csup\u003e24\u003c/sup\u003e Meanwhile, the FORTE category concept is customizable and transferable across various medical image captioning tasks by substituting categorical keyword bank contents. (Extended Data Fig. 2)\u003c/p\u003e\n\u003cp\u003eHuman expert evaluation in natural language processing experiments has been conducted under distinct experiment designs and served diverse study purposes. Therefore, the resulting perspectives were often inconsistent and non-comparable across individual contexts. To this end, related works had instrumented quantitative (completeness, correctness, conciseness) and qualitative (content, language, structure) measurements to dissect the riveting characters that discern synthesized clinical reports from human reports.\u003csup\u003e35-37\u003c/sup\u003e By adopting similar designs with objective linguistic criteria, we found that both reviewer success rate and answer alternation reasons (“Suspicious wording” and “Both not mentioning critical features”) were associated with the writing style (“Familiarity and voice” and \"Specificity or vagueness of details\") rather than the sentence-level writing quality and coherence.\u003csup\u003e28\u003c/sup\u003e The significance of medical report writing style was also underlined in an independent prompting study.\u003csup\u003e36\u003c/sup\u003e Interestingly, we observed that input case imbalance can influence caption writing style, potentially related to over-fitting noticed in general model training (see also Extended Data Fig. 3).\u003c/p\u003e\n\u003cp\u003eThis study has several limitations, which should be addressed in future works. First, this pilot brain CT study has no counterpart MLLM module to benchmark, and hence we cannot justify the efficacy at a SOTA level; however, we applied external validation to ensure the caption validity in the brain CT module. Second, BrainGPT is trained on degeneration-oriented data and thus it fails to caption the malignancy tumor and acute traumatic features in CQ500. This phenomenon reflects that the training material may prime the dexterity of the resultant module.\u003csup\u003e38\u003c/sup\u003e Hence, we suggest enrolling diverse disease etiologies with the aim of differential diagnosis to improve MLLM generalization on border brain CT features. Last, we conducted CVIT and invented clinical-oriented evaluations (sentence-pairing, negation removal, and FORTE), but we did not experiment on whether changing the model backbone could benefit brain CT captioning. Future research avenues could be comparing multi-model results and fine-tuning the vision encoder and the language model for CT.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eCode and data availability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll code used for experiments in this study can be found in a GitHub repository (https://github.com/charlierabea/FORTE), which also contains a model weight link to our best instruction-tuned model (BrainGPT-keyword). This study used data from Taipei Veterans General Hospital (TPEVGH) for the training process and CQ500 from Qure.ai for external validation.\u003csup\u003e39\u003c/sup\u003e Data from TPEVGH cannot be released due to IRB regulations, but researchers can access CQ500 via the provided reference. Note that any further distribution of datasets is subject to the terms of use and data-sharing agreements stipulated by the original creators.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFunded by grants of the Taiwan National Science and Technology Council (NSTC 112-2321-B-A49-007, NSTC 1 11-2320-B-A49-028-MY3, NSTC 112-2124-M-038-001, and NSTC 112-2314-B-032-001) and Taipei Veterans G eneral Hospital (V112C-026 and 112VACS-007)\u003c/p\u003e\n\u003cp\u003eThis study is based in part on data from the Big Data Center, Taipei Veterans General Hospital (BDC, TPEVGH). The interpretations and conclusions contained herein do not represent the position of Taipei Veterans General Hospital.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor\u003c/strong\u003e\u003cstrong\u003eContributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eC.Y.L collected data, designed visual instruction tuning instructions, ran experiments, analyzed results, created figures and wrote the manuscript. K.J.C proposed the FORTE framework, analyzed results, and wrote the manuscript. C.F.Y assisted in model fine-tuning and designed sentence pairing. H.Y.W conducted qualitative analysis and created figures. W.T.C assisted in model fine-tuning and provided technical advice. H.B. assisted in model survey and academic writing. L.C, Y.P.Y, Y.C.C, and S.P.C advised on study design and provided meaningful feedback. J.F.L directed the Turing Test and provided feedback on instruction design and the FORTE framework as a radiologist. K.W.C and S.H.C guided the project and provided critical suggestions for overall direction. All authors contributed to the interpretation and provided critical feedback on the analyses and manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting Interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare no competing interests.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n \u003cli\u003eCao, K.\u003cem\u003e, et al.\u003c/em\u003e Large-scale pancreatic cancer detection via non-contrast CT and deep learning. \u003cem\u003eNat Med\u003c/em\u003e \u003cstrong\u003e29\u003c/strong\u003e, 3033-3043 (2023).\u003c/li\u003e\n \u003cli\u003eGroh, M.\u003cem\u003e, et al.\u003c/em\u003e Deep learning-aided decision support for diagnosis of skin disease across skin tones. \u003cem\u003eNat Med\u003c/em\u003e \u003cstrong\u003e30\u003c/strong\u003e, 573-583 (2024).\u003c/li\u003e\n \u003cli\u003eTian, F.\u003cem\u003e, et al.\u003c/em\u003e Prediction of tumor origin in cancers of unknown primary origin with cytology-based deep learning. \u003cem\u003eNature Medicine\u003c/em\u003e (2024).\u003c/li\u003e\n \u003cli\u003eDai, L.\u003cem\u003e, et al.\u003c/em\u003e A deep learning system for predicting time to progression of diabetic retinopathy. \u003cem\u003eNat Med\u003c/em\u003e \u003cstrong\u003e30\u003c/strong\u003e, 584-594 (2024).\u003c/li\u003e\n \u003cli\u003eRajpurkar, P. \u0026amp; Lungren, M.P. The Current and Future State of AI Interpretation of Medical Images. \u003cem\u003eN Engl J Med\u003c/em\u003e \u003cstrong\u003e388\u003c/strong\u003e, 1981-1990 (2023).\u003c/li\u003e\n \u003cli\u003eBoiselle, P.M. Computed tomography screening for lung cancer. \u003cem\u003eJAMA\u003c/em\u003e \u003cstrong\u003e309\u003c/strong\u003e, 1163-1170 (2013).\u003c/li\u003e\n \u003cli\u003eWysoki, M.G.\u003cem\u003e, et al.\u003c/em\u003e Head trauma: CT scan interpretation by radiology residents versus staff radiologists. \u003cem\u003eRadiology\u003c/em\u003e \u003cstrong\u003e208\u003c/strong\u003e, 125-128 (1998).\u003c/li\u003e\n \u003cli\u003eBoag, W.\u003cem\u003e, et al.\u003c/em\u003e Baselines for chest x-ray report generation. in \u003cem\u003eMachine learning for health workshop\u003c/em\u003e 126-140 (PMLR, 2020).\u003c/li\u003e\n \u003cli\u003eChen, Z., Song, Y., Chang, T.-H. \u0026amp; Wan, X. Generating radiology reports via memory-driven transformer. \u003cem\u003earXiv preprint arXiv:2010.16056\u003c/em\u003e (2020).\u003c/li\u003e\n \u003cli\u003eSelivanov, A.\u003cem\u003e, et al.\u003c/em\u003e Medical image captioning via generative pretrained transformers. \u003cem\u003eScientific Reports\u003c/em\u003e \u003cstrong\u003e13\u003c/strong\u003e, 4171 (2023).\u003c/li\u003e\n \u003cli\u003eYang, S.\u003cem\u003e, et al.\u003c/em\u003e Radiology report generation with a learned knowledge base and multi-modal alignment. \u003cem\u003eMedical Image Analysis\u003c/em\u003e \u003cstrong\u003e86\u003c/strong\u003e, 102798 (2023).\u003c/li\u003e\n \u003cli\u003eJohnson, A.E.W.\u003cem\u003e, et al.\u003c/em\u003e MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. \u003cem\u003eSci Data\u003c/em\u003e \u003cstrong\u003e6\u003c/strong\u003e, 317 (2019).\u003c/li\u003e\n \u003cli\u003eDemner-Fushman, D.\u003cem\u003e, et al.\u003c/em\u003e Preparing a collection of radiology examinations for distribution and retrieval. \u003cem\u003eJ Am Med Inform Assoc\u003c/em\u003e \u003cstrong\u003e23\u003c/strong\u003e, 304-310 (2016).\u003c/li\u003e\n \u003cli\u003eChen, W., Shen, L., Li, X. \u0026amp; Yuan, Y. Fine-Grained Image-Text Alignment in Medical Imaging Enables Cyclic Image-Report Generation. arXiv:2312.08078 (2023).\u003c/li\u003e\n \u003cli\u003eLi, C.\u003cem\u003e, et al.\u003c/em\u003e LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day. arXiv:2306.00890 (2023).\u003c/li\u003e\n \u003cli\u003eTu, T.\u003cem\u003e, et al.\u003c/em\u003e Towards Generalist Biomedical AI. \u003cem\u003eNEJM AI\u003c/em\u003e \u003cstrong\u003e1\u003c/strong\u003e, AIoa2300138 (2024).\u003c/li\u003e\n \u003cli\u003eHaydel, M.J.\u003cem\u003e, et al.\u003c/em\u003e Indications for computed tomography in patients with minor head injury. \u003cem\u003eN Engl J Med\u003c/em\u003e \u003cstrong\u003e343\u003c/strong\u003e, 100-105 (2000).\u003c/li\u003e\n \u003cli\u003eK\u0026ouml;nig, M. Brain perfusion CT in acute stroke: current status. \u003cem\u003eEuropean Journal of Radiology\u003c/em\u003e \u003cstrong\u003e45\u003c/strong\u003e, S11-S22 (2003).\u003c/li\u003e\n \u003cli\u003eBanerjee, S. \u0026amp; Lavie, A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. in \u003cem\u003eProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization\u003c/em\u003e 65-72 (2005).\u003c/li\u003e\n \u003cli\u003ePapineni, K., Roukos, S., Ward, T. \u0026amp; Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. in \u003cem\u003eProceedings of the 40th annual meeting of the Association for Computational Linguistics\u003c/em\u003e 311-318 (2002).\u003c/li\u003e\n \u003cli\u003eLin, C.-Y. Rouge: A package for automatic evaluation of summaries. in \u003cem\u003eText summarization branches out\u003c/em\u003e 74-81 (2004).\u003c/li\u003e\n \u003cli\u003eSantos, G.O.d., Colombini, E.L. \u0026amp; Avila, S. CIDEr-R: Robust consensus-based image description evaluation. \u003cem\u003eProceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021)\u003c/em\u003e (2021).\u003c/li\u003e\n \u003cli\u003eLiu, G.\u003cem\u003e, et al.\u003c/em\u003e Clinically Accurate Chest X-Ray Report Generation. arXiv:1904.02633 (2019).\u003c/li\u003e\n \u003cli\u003eIrvin, J.\u003cem\u003e, et al.\u003c/em\u003e CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. \u003cem\u003eProceedings of the AAAI Conference on Artificial Intelligence\u003c/em\u003e \u003cstrong\u003e33\u003c/strong\u003e, 590-597 (2019).\u003c/li\u003e\n \u003cli\u003eLiu, H., Li, C., Wu, Q. \u0026amp; Lee, Y.J. Visual Instruction Tuning. arXiv:2304.08485 (2023).\u003c/li\u003e\n \u003cli\u003eLi, B.\u003cem\u003e, et al.\u003c/em\u003e Otter: A Multi-Modal Model with In-Context Instruction Tuning. arXiv:2305.03726 (2023).\u003c/li\u003e\n \u003cli\u003eSinghal, K.\u003cem\u003e, et al.\u003c/em\u003e Large language models encode clinical knowledge. \u003cem\u003eNature\u003c/em\u003e \u003cstrong\u003e620\u003c/strong\u003e, 172-180 (2023).\u003c/li\u003e\n \u003cli\u003eCasal, J.E. \u0026amp; Kessler, M. Can linguists distinguish between ChatGPT/AI and human writing?: A study of research ethics and academic publishing. \u003cem\u003eResearch Methods in Applied Linguistics\u003c/em\u003e \u003cstrong\u003e2\u003c/strong\u003e, 100068 (2023).\u003c/li\u003e\n \u003cli\u003eEthem Hamamci, I., Er, S. \u0026amp; Menze, B. CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging. arXiv:2403.06801 (2024).\u003c/li\u003e\n \u003cli\u003eMiura, Y., Zhang, Y., Tsai, E.B., Langlotz, C.P. \u0026amp; Jurafsky, D. Improving factual completeness and consistency of image-to-text radiology report generation. \u003cem\u003earXiv preprint arXiv:2010.10042\u003c/em\u003e (2020).\u003c/li\u003e\n \u003cli\u003eNicolson, A., Dowling, J. \u0026amp; Koopman, B. Improving chest X-ray report generation by leveraging warm starting. \u003cem\u003eArtificial Intelligence in Medicine\u003c/em\u003e \u003cstrong\u003e144\u003c/strong\u003e, 102633 (2023).\u003c/li\u003e\n \u003cli\u003eYang, L.\u003cem\u003e, et al.\u003c/em\u003e Advancing Multimodal Medical Capabilities of Gemini. arXiv:2405.03162 (2024).\u003c/li\u003e\n \u003cli\u003eTouvron, H.\u003cem\u003e, et al.\u003c/em\u003e LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 (2023).\u003c/li\u003e\n \u003cli\u003eAbbasian, M.\u003cem\u003e, et al.\u003c/em\u003e Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI. \u003cem\u003enpj Digital Medicine\u003c/em\u003e \u003cstrong\u003e7\u003c/strong\u003e, 82 (2024).\u003c/li\u003e\n \u003cli\u003eVan Veen, D.\u003cem\u003e, et al.\u003c/em\u003e Adapted large language models can outperform medical experts in clinical text summarization. \u003cem\u003eNature Medicine\u003c/em\u003e (2024).\u003c/li\u003e\n \u003cli\u003eYan, B.\u003cem\u003e, et al.\u003c/em\u003e Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting. arXiv:2310.17811 (2023).\u003c/li\u003e\n \u003cli\u003eBoag, W., Kan\u0026eacute;, H., Rawat, S., Wei, J. \u0026amp; Goehler, A. A Pilot Study in Surveying Clinical Judgments to Evaluate Radiology Report Generation. in \u003cem\u003eProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency\u003c/em\u003e 458\u0026ndash;465 (Association for Computing Machinery, Virtual Event, Canada, 2021).\u003c/li\u003e\n \u003cli\u003eYoussef, A.\u003cem\u003e, et al.\u003c/em\u003e External validation of AI models in health should be replaced with recurring local validation. \u003cem\u003eNature Medicine\u003c/em\u003e \u003cstrong\u003e29\u003c/strong\u003e, 2686-2687 (2023).\u003c/li\u003e\n \u003cli\u003eChilamkurthy, S.\u003cem\u003e, et al.\u003c/em\u003e Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. \u003cem\u003eThe Lancet\u003c/em\u003e \u003cstrong\u003e392\u003c/strong\u003e, 2388-2396 (2018).\u003c/li\u003e\n \u003cli\u003eLin, T.-Y.\u003cem\u003e, et al.\u003c/em\u003e Microsoft COCO: Common Objects in Context. in \u003cem\u003eComputer Vision \u0026ndash; ECCV 2014\u003c/em\u003e (eds. Fleet, D., Pajdla, T., Schiele, B. \u0026amp; Tuytelaars, T.) 740-755 (Springer International Publishing, Cham, 2014).\u003c/li\u003e\n \u003cli\u003eZhang, T., Kishore, V., Wu, F., Weinberger, K.Q. \u0026amp; Artzi, Y. Bertscore: Evaluating text generation with bert. \u003cem\u003earXiv preprint arXiv:1904.09675\u003c/em\u003e (2019).\u003cstrong\u003e\u003c/strong\u003e\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Method","content":"\u003cp\u003e\u003cstrong\u003eStudy Design and Oversight\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIn this study, we trained BrainGPT to generate 3D brain CT reports. Then, we examined the caption efficacy by (1) adapting traditional evaluation metrics (2) proposing a clinical-oriented FORTE metric, (3) applying an external validation CQ500, and (4) conducting the linguistic-embedded \u003cem\u003eTuring\u003c/em\u003e test.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStudy Patient\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe collected 18,885 brain CT scans (742,501 slices) from 9,689 patients with Alzheimer's Disease (mean age = 82.59 years [standard deviation = 9.3 years]; 56.4% male) at Taipei Veterans General Hospital in Taipei, Taiwan, between January 1, 2010, and December 31, 2022. All data were collected under institutional review board approval (2023-10-002 BC). Informed consent was exempted due to the retrospective nature of the data collection and the use of deidentified CT images. The CT images included a variety of common neurology conditions affecting the skull, brain parenchyma, nasal sinuses, and the eye, and were collected by radiologists who routinely obtain CT images and write image reports based on the images and the patient's medical records. Since Alzheimer's Disease is a progressive degenerative condition predominantly seen in the elderly, the dataset includes images of normal brains, past infarcts that still show manifestations, chronic brain conditions, and acute brain lesions.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eClinical Visual Instruction Tuning (CVIT)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo address the domain constraints of standard MLLM, we conducted multiple end-to-end visual instruction tuning processes on the multi-image mode Otter foundation model, enhancing its applicability to brain CT radiology features.\u003csup\u003e25,26\u003c/sup\u003e Based on the Flamingo model, the Otter paradigm connects the LLaMA-7B language encoder and the frozen CLIP ViT-L/14 vision encoder via a trainable perceiver resampler module and multiple cross-attention layers inserted into the LLaMA-7B architecture. Within the original LLaMA-7B structure, all modules except for the input/output embeddings were frozen to reduce training costs. The training duration for each resulting model was 12 hours on two NVIDIA A100 GPUs, achieving 3 epochs.\u003c/p\u003e\n\u003cp\u003eTo facilitate multi-image in-context learning capacity, we formulated the data into image-instruction-answer triplets, with the instruction tokenized and the image enhanced prior to input into the model. We designed four distinct fine-tuning conditions including regular visual instruction tuning (RVIT, Plain Instruction, and In-context example Instruction) and clinical visual instruction tuning (CVIT, Template Instruction, and Keyword Instruction), each corresponding to an adherence hierarchy to clinical essence. We named the final instruction-tuned models BrainGPT-plain, BrainGPT-example, BrainGPT-template, and BrainGPT-keyword.\u003c/p\u003e\n\u003cp\u003eFor each visual instruction tuning process, BrainGPT-plain was fine-tuned using plain instruction, conveying the model’s role as a radiology assistant; BrainGPT-example was fine-tuned using in-context example instruction, adopting a 3-shot example approach due to available RAM constraints as based on the work of Singhal et al.;\u003csup\u003e27\u003c/sup\u003e BrainGPT-template was fine-tuned using template instruction, involving a structured and predefined set of questions or points that need to be addressed; BrainGPT-keyword was fine-tuned using keyword instruction, focusing on essential areas or categorical guidelines that direct the model’s response generation process. Detailed instruction examples can be referenced in Extended Data Fig. 4.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDataset preparation\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTraining Dataset.\u003c/strong\u003e Since the Otter architecture requires image-text pair instances to be of the same size (24 slices), we sampled 365,928 slices from 15,238 scans representing 7,747 patients from a total of 597,335 slices from 15,247 scans representing 7,751 patients for the training process. The system was then tested on 87,312 slices sampled from 145,166 slices of 3,638 CT scans representing 1,938 patients.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eExternal Validation Dataset.\u003c/strong\u003e\u0026nbsp; The CQ500 dataset, consisting of 1,154 available CT scans from 491 patients, was downloaded from the Qure.ai website.\u003csup\u003e39\u003c/sup\u003e The dataset focuses on image features such as brain parenchyma (plain scans), bone (bone scans), and blood vessels (post-contrast scans). Only non-contrast CT scans with slice numbers between 23 and 40 were selected to build the external validation dataset (n=133). This ensures that slice thickness and details are similar to our training dataset and fit in the Otter framework. Ground truth was based on a read.csv file from the CQ500 dataset, and the majority rule was applied among the three raters to summarize the “Mass effect,” “Hemorrhage event,” and “Midline shift” labels.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFeature-Oriented Radiology Task Evaluation (FORTE)\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe proposed the Feature-Oriented Radiology Task Evaluation (FORTE) framework to capture radiology keywords in terms of Degree (size and intensity), Landmark (location and anatomy), Feature (disease traits and primary findings), and Impression (final diagnosis) of the disease. (For details on the keyword list, see Extended Data Table 7). This list is not just a compilation of keywords but also includes synonyms, allowing the system to recognize a broader array of related terms that may appear in different contexts, addressing the challenge of lexical variability inherent in clinical reports. The F1 score is calculated for each category, providing a multi-faceted evaluation of the system's performance. Additionally, we compute Pearson’s correlation coefficient (Pearson’s r) for each FORTE category with traditional metrics, offering a deeper understanding of their applicability and limitations in radiological report evaluation.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTraditional metrics\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe compared the clinical evaluation fitness of FORTE against the standard similarity-based evaluation metrics, including BLEU (Bilingual Evaluation Understudy, set range 0-100), METEOR (Metric for Evaluation of Translation with Explicit ORdering, set range 0-100), ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation with Longest Common Subsequence, set range 0-100), and CIDEr-R (Robust Consensus-based Image Description Evaluation, set range 0-1000) (Extended Data Table 8)\u003csup\u003e19-22\u003c/sup\u003e All standard evaluations were executed using the Microsoft Common Objects in Context (MS-COCO) toolkit.\u003csup\u003e40\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003e\u0026nbsp; \u0026nbsp; Additionally, to address the list-by-list structure, diverse paraphrasing, and differential diagnosis-oriented negation description of brain CT reports, we incorporated sentence pairing (inspired by cosine similarity calculation in BERTScore)\u003csup\u003e41\u003c/sup\u003e and negation removal before applying the evaluation formulas. By these means, sentence pairing releases the sequential constraints of disease descriptors, and negation removal reduces the false positives in evaluation reports. Specifically, the reports were embedded and vectorized using the all-mpnet-base-v2 model from the SentenceTransformer library before pairing and scoring.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLinguistic-embedded\u003c/strong\u003e\u003cem\u003e\u0026nbsp;\u003cstrong\u003eTuring\u0026nbsp;\u003c/strong\u003e\u003c/em\u003e\u003cstrong\u003etest\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo examine whether the BrainGPT CT report recapitulates the linguistic texture of radiologist reports, we conducted a \u003cem\u003eTuring\u003c/em\u003e test involving physicians and radiologists. Each participant was asked to distinguish BrainGPT reports from radiologist reports. The study was structured around four measuring axes:\u003c/p\u003e\n\u003cp\u003e(1) \u003cem\u003eTuring\u003c/em\u003e test: Can physicians tell the difference between BrainGPT reports and radiologist reports?\u003c/p\u003e\n\u003cp\u003e(2) Confidence rate: How confident are the reviewers in their ratings?\u003c/p\u003e\n\u003cp\u003e(3) Inter-leaved dependency: Do physicians alter their assessments and confidence rates after reviewing the original CT scans?\u003c/p\u003e\n\u003cp\u003e(4) Linguistic criteria: What is the linguistic rationale behind physicians' impressions?\u003csup\u003e28\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eTo explore the aforementioned questions, we collected survey and semi-structured rationale interview data. The physician survey was composed of six caption pairs, each comprising a BrainGPT report and a radiologist report. These examples included diverse disease instances including lacunar infarct, subdural hemorrhage, brain atrophy, and midline (cerebral falx) shift, thereby encompassing a range of both acute and chronic cerebral alterations for expert evaluation. (Details can be referred to Extended Data Fig. 5)\u003c/p\u003e"},{"header":"Supplementary Table","content":"\u003cp\u003eSupplementary Table 1 is not available with this version\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Multi-modal Large Language Model (MLLM), Three-dimensional (3D) brain computed tomography (CT), Visual instruction tuning, Turing test, Clinical Efficacy (CE) Evaluation Metric","lastPublishedDoi":"10.21203/rs.3.rs-4558754/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4558754/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eMulti-modal large language models (MLLMs) have been given free rein to explore exciting medical applications with a primary focus on radiology report generation. Nevertheless, the preliminary MLLM successful attempts in 2D medical image-text pair captioning are incompetent to reflect the real-world diagnostic challenge in the volumetric 3D anatomy. Toward deploying MLLM for more applicable diagnostic context, we noticed that the (1) scarcity of 3D image training dataset, (2) the direct use of undifferentiated foundation MLLMs, and (3) the lack of pertinent caption evaluation metrics were independent domain-specific constraints that integratively hobbles the iteration of next-generation medical MLLM research. In this regard, this study collected a 3D-BrainCT dataset (18,885 text-scan pairs) and applied clinical visual instruction tuning (CVIT) to train volumetric anatomy-sensible BrainGPT models to generate radiology-adherent 3D brain CT reports. Statistically, our BrainGPT model scored BLEU-1 = 44.35, BLEU-4 = 20.38, METEOR = 30.13, ROUGE-L = 47.6, and CIDEr-R = 211.77 during internal testing and demonstrated an accuracy of 0.91 in captioning midline shifts on the external validation CQ500 dataset. By further inspecting the captioned report, we reported that the traditional metrics appeared to measure only the surface text similarity and failed to gauge the information density of the diagnostic purpose. To close this gap, we proposed a novel Feature-Oriented Radiology Task Evaluation (FORTE) to estimate the clinical relevance (lesion feature and landmarks) of the report. Notably, the BrainGPT model scored an average FORTE 0.71 F1-score (degree=0.661; landmark=0.706; feature=0.693, and impression=0.779). To demonstrate that BrainGPT models possess objective readiness to generate human-like radiology reports, we conducted a Turing test that enrolled 11 physician evaluators, and around 74% of the BrainGPT-generated captions were indistinguishable from those written by humans. While various computational intelligence researchers have advocated the avant-garde MLLM applications, our work embodies a holistic framework that showcased the first-hand experience of curating a 3D brain CT dataset, fine-tuning anatomy-sensible language models, and proposing robust radiology evaluation metrics. We deemed that the adventure of docking MLLM for 3D brain CT report generation may unfold new MLLM applications at the forefront of human-machine collaborated modern healthcare.\u003c/p\u003e","manuscriptTitle":"Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-07-03 19:13:03","doi":"10.21203/rs.3.rs-4558754/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-communications","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"NCOMMS","sideBox":"Learn more about [Nature Communications](http://www.nature.com/ncomms/)","snPcode":"","submissionUrl":"https://mts-ncomms.nature.com/","title":"Nature Communications","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Communications","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"2463636b-a579-459a-8c11-1854b2372c7f","owner":[],"postedDate":"July 3rd, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":33093411,"name":"Biological sciences/Computational biology and bioinformatics/Machine learning"},{"id":33093412,"name":"Health sciences/Health care/Medical imaging/Three-dimensional imaging"}],"tags":[],"updatedAt":"2025-03-07T08:07:49+00:00","versionOfRecord":{"articleIdentity":"rs-4558754","link":"https://doi.org/10.1038/s41467-025-57426-0","journal":{"identity":"nature-communications","isVorOnly":false,"title":"Nature Communications"},"publishedOn":"2025-03-06 05:00:00","publishedOnDateReadable":"March 6th, 2025"},"versionCreatedAt":"2024-07-03 19:13:03","video":"","vorDoi":"10.1038/s41467-025-57426-0","vorDoiUrl":"https://doi.org/10.1038/s41467-025-57426-0","workflowStages":[]},"version":"v1","identity":"rs-4558754","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4558754","identity":"rs-4558754","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}