OmiMRI: A Clinical-adaptive AI Framework for Format-Free Interpretation of Heterogeneous Brain MRIs

OmiMRI: A Clinical-adaptive AI Framework for Format-Free Interpretation of Heterogeneous Brain MRIs | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article OmiMRI: A Clinical-adaptive AI Framework for Format-Free Interpretation of Heterogeneous Brain MRIs Lei Ma, Feng Su, Xiaoping Yi, Ye Cheng, Yongjie Ma, Zeming Tan, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8651933/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted You are reading this latest preprint version Abstract Clinical brain MRI analysis faces a fundamental challenge: bridging the gap between oversimplified research developments and the inherent heterogeneity of real-world clinical practice. Quantifying this gap, our analysis of 26 MRI attributes across 22 clinical datasets reveals substantial heterogeneity across institutions and patients. Current AI tools typically require rigid input formats, necessitating extensive data exclusion or preprocessing that severely limits their real-world utility. Here we present OmiMRI, a unified, format-free framework designed to bridge this gap by enabling adaptive processing of arbitrary MRI combinations. Rather than strictly defining a standalone architecture, OmiMRI functions as a universal framework that integrates diverse pretrained 2D/3D convolutional and Transformer-based networks as feature encoders. Through a self-attention mechanism and dynamic weighting to fuse features from variable inputs, OmiMRI decouples clinical performance from rigid input specifications and enables adaptive processing of arbitrary MRI combinations. Across 15 diverse classification, segmentation, and regression tasks, OmiMRI demonstrates robust input-scaling capabilities, yielding significant improvements over traditional fixed-input models. Notably, OmiMRI outperforms advanced medical imaging foundation models (e.g., BrainIAC and BrainSegFounder) in 94.4% of comparisons involving 2–4 input MRIs under consistent experimental conditions. Furthermore, the framework exhibits continuous performance gains through the incremental incorporation of multi-center data. In a clinically demanding, data-limited task distinguishing glioblastoma from metastasis, OmiMRI achieved diagnostic performance matching or exceeding that of senior neuroradiologists (AUROC 0.931 vs. 0.907, P > 0.05; AUPRC 0.973 vs. 0.930, P < 0.05), while providing interpretable attention maps aligned with radiological landmarks. Together, these results establish OmiMRI as a clinically adaptive AI paradigm that transform format-rigid modeling into flexible, expert-level systems capable of embracing the heterogeneity of real-world patient data. Biological sciences/Computational biology and bioinformatics/Image processing Biological sciences/Biological techniques/Imaging/Magnetic resonance imaging Physical sciences/Mathematics and computing/Computational science Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Introduction Magnetic resonance imaging (MRI) is a powerful, non-invasive technique for visualizing both the structural and functional details of the brain 1–3 . It remains the gold standard for diagnosing numerous neurological disorders 4–6 , including glioblastoma (GBM), brain metastases (MET), cerebrovascular diseases, neurodegenerative disorders, neuropsychiatric disorders, and pathological aging. Among these, GBM and MET are the most prevalent malignant brain tumors worldwide, with a median survival time of less than 16 months 7–9 , posing significant challenges for personalized diagnosis. Artificial intelligence (AI) holds the potential to fully leverage the value of MRIs. However, most existing AI models are still in the developmental stage and face an "AI chasm" before they can be effectively deployed in clinical practice 10–12 . Despite remarkable progress, most AI models implicitly assume that heterogeneity in clinical MRI should be minimized or corrected through standardization or harmonization 13,14 . This assumption fundamentally conflicts with real-world clinical practice, where variability is intrinsic, informative, and often unavoidable. To fully realize the potential of AI models, efficient data and robust frameworks are essential. MRI plays a crucial role in modern healthcare, with approximately 100 to 150 million scans performed annually worldwide by the 2020s 15–17 . This has led to the accumulation of vast quantities of clinical brain MRIs, including numerous open-source datasets available through platforms such as The Cancer Imaging Archive (TCIA) 18 , NeuroImaging Tools & Resources Collaboratory (NITRC) 19 , UK Biobank 20 , and OpenNeuro 21 . These resources hold immense value for data-driven AI models aimed at enhancing the neuroimage analysis and neurological disorders diagnosis 22–24 . Clinical MRIs are extremely heterogeneous and inherently personalized, exhibiting significant variations in scan sequences, slice resolution, spacing, and slice count. However, most advanced brain MRI models remain confined to oversimplified settings with meticulously preprocessed data that adhere to strict formatting requirements (Fig. S1). The notable rigidity of these models prevents them from effectively handling the substantial heterogeneity observed in clinical settings 25,26 . Despite efforts to address issues such as site heterogeneity and modality missing 13,14,27,28 , a substantial gap remains between research development and clinical deployment 10–12 . Instead of enforcing a canonical input format, we propose that clinically meaningful AI systems should adapt to whatever data are available for each individual patient. To bridge this gap, here we present OmiMRI, a unified, attribute-agnostic framework capable of processing arbitrary combinations of MRI inputs (Fig. 1, Fig. S2). Unlike conventional "format-rigid" approaches that require strict input standardization which often necessitating data exclusion or extensive preprocessing, OmiMRI adopts a "format-free" paradigm designed for direct training and inference on highly variable clinical data. OmiMRI supports various advanced backbones for initial feature extraction, followed by self-attention mechanisms or dynamic weighting to fuse variable-length feature representations. When provided with two or more combined MRIs, OmiMRI consistently outperforms conventional single-input models across 15 brain MRI tasks, with performance improves steadily as more sequences are incorporated. We assembled an unprecedentedly large and diverse multi-center brain MRI cohort, integrating nearly all publicly available brain tumor and cerebrovascular disease resources alongside a proprietary dataset spanning 16 hospitals. The in-house data cover over 88% of total patient cases for four of the six major brain diseases, substantially enriching resources for high-quality brain MRI research. Through continual learning experiments, we rapidly established a model prototype using single-center data, then progressively expanded the dataset to include more centers and samples. This enabled iterative version evolution, with performance improving across external validation datasets. In a challenging data-limited clinical task, OmiMRI matches or surpasses senior neuroradiologists in discriminating GBM from brain MET, demonstrating robust performance and clinically interpretable attention patterns. Together, these results establish OmiMRI as a practical, scalable paradigm for clinically adaptive brain MRI analysis under real-world heterogeneity. Results OmiMRI: a practical AI framework for heterogeneous clinical environments OmiMRI is specifically designed to address the challenges posed by the extreme heterogeneity of MRIs encountered in clinical practice. This AI framework is "format-free" and effectively handles arbitrary MRI inputs, accommodating significant variations in MRI count, sequence combinations, and imaging properties (Fig. 1A, Fig. 2A). The complexity of clinical MRIs arises from three primary sources. First, there is a broad array of MRI sequences routinely used, with over 12 common types, including T1-weighted (T1w), T2-weighted (T2w), contrast-enhanced T1 (T1c), T2 fluid-attenuated inversion recovery (T2f), magnetic resonance angiography (MRA), and apparent diffusion coefficient (ADC). Second, acquisition parameters vary considerably across scans, with differences in scan axis, slice resolution, slice spacing, and slice count (Fig. 2A). Finally, clinical practice introduces inherent combinatorial complexity, as MRI examinations are tailored to individual diagnostic needs, resulting in significant inter-patient variability (Fig. 2A). Conventional AI strategies typically employ "format-rigid" modeling, which requires standardized inputs and consequently forcing the exclusion of non-conforming data (Fig. 1B). They also necessitate training distinct models for every unique combination of input modalities (Fig. 1C). In contrast, OmiMRI introduces a "format-free" modeling paradigm designed to directly process arbitrary, heterogeneous MRI combinations (Fig. 1B). OmiMRI uses a single architecture capable of handling varying input subsets, ranging from single T1w scans to complex multi-modal combinations. This design significantly reduces preprocessing labor and enhances practical data utility compared to conventional methods (Fig. 1C). OmiMRI functions as a universal interface that supports a series of pretrained backbones (such as BrainSegFounder, BrainIAC, or standard convolutional neural networks (CNNs)) for initial feature extraction. To address the dimensionality mismatch caused by heterogeneous inputs, OmiMRI projects the extracted features into a normalized space prior to fusion (Fig. 1C). The system then applies task-specific aggregation strategies—using Transformer-based self-attention for linear head tasks (classification and regression) or dynamic weighting for dense prediction tasks (segmentation)—enabling a single, scalable platform to handle diverse brain MRI tasks across institutions. Quantifying the heterogeneity of clinical brain MRIs Clinical brain MRI data exhibit significant heterogeneity compared to research-centered datasets, primarily due to variations in scanning protocols driven by diagnostic and treatment requirements. In research settings, MRI acquisition protocols are designed with specific research objectives, often simplifying the scanning and analysis process by tightly controlling MRI parameters. Consequently, research-driven MRI data are typically structured and standardized, showing high consistency across subjects (Fig. 2A). However, MRI datasets from different research centers or with varying objectives often display substantial heterogeneity, reflecting differences in scanning and preprocessing protocols. In contrast, the primary goal of clinical brain MRI is to diagnose and treat diseases. The selection of scanning sequences, counts, and parameters is guided by clinical needs. Both six open-source datasets and in-house brain MRI data from 16 hospitals show extreme heterogeneity (Fig. 2A). This heterogeneity is evident in two aspects. First, there is significant individual variation; each patient's disease condition is unique, making clinical brain MRI data inherently personalized. Second, there is substantial variability across institutions and disease types. Different brain diseases often require distinct combinations of MRI sequences to accurately reveal disease status. Additionally, scanning protocols vary widely among institutions. Even widely used MRI sequences, such as T1w and T2w, exhibit considerable complexity in clinical practice. For example, in the RocklandSample research dataset, T1w data demonstrate perfectly consistent slice parameters—including view, resolution, number, and step—across all subjects (Fig. 2B). However, in clinical practice, such as in the HunanXiangya dataset, T1w and T2w slice parameters can vary across as many as 120 different combinations (Fig. 2B, C). There are also substantial differences across centers and diseases; for instance, the T1w scan slice parameters in the ABIDE-1 dataset show 12 combinations, while the T2w scan slice parameters in the YunnanRenmin1st dataset exhibit 24 different combinations. Most importantly, the vast majority of brain MRIs are generated in clinical settings, providing more abundant, valuable, and complicated resources than curated research datasets. The immense potential of clinical brain MRI resources remains untapped, highlighting the need for AI frameworks capable of addressing their extreme heterogeneity. Performance of OmiMRI and comparison with classic models To evaluate OmiMRI's capability to handle arbitrary combinations of input sequences and achieve effective feature fusion, we progressively increased the number of MRI sequences used during testing. Initially, only T1w were used, followed by the addition of T2w, T1c, and T2f sequences. In volume and image classification tasks (LGG vs. HGG), the AUROC increased significantly as more MRI sequences were incorporated (Fig. 3A, B). Similarly, in segmentation tasks, the mean Dice score for each label showed significant improvement with the inclusion of additional sequences (Fig. 3C, D; Fig. S3). To evaluate OmiMRI’s versatility, we tested the model across a broad spectrum of MRI tasks. These tasks were grouped into three main categories: volume classification and regression (six tasks, Fig. 3E), image segmentation (six tasks, Fig. 3F), and image classification (three tasks, Fig. 3G). In traditional AI frameworks, each model is typically tailored to a specific task type. For example, the Conv3D TPN model is used for volume classification and regression, while ResNet50 is used for image classification. In contrast, the OmiMRI framework demonstrated its versatility by successfully handling 15 tasks across these three categories, highlighting its broad applicability. For each task, we trained models using various combinations of MRI inputs. During testing, we categorized the results based on the number of MRIs: OmiMRI_I1, OmiMRI_I2, and OmiMRI_I3+ represent the same model using combined inputs with 1, 2, and three or more MRIs for inference, respectively. The results showed that performance consistently improved as the number of input MRIs increased across all downstream tasks. Compared to the nine classical models that accept fixed input with a single MRI, OmiMRI_I2 and OmiMRI_I3+ exhibited a comprehensive performance advantage (Fig. 3E-G). We further demonstrate that OmiMRI outperforms fixed-input models in both performance and practicality. Using identical backbone architectures, datasets, and training conditions, OmiMRI consistently surpasses four medical imaging foundation models as well as two natural video‑pretrained models when processing 2 to 4 input MRIs. Compared to fixed-input models, OmiMRI achieves superior F1 scores in 94.4% of comparisons, with an average absolute improvement of 0.038 ± 0.031 (Table 1, Table S1). These results confirm OmiMRI's "attribute-agnostic" nature and its ability to integrate diverse MRI inputs, thereby enhancing diagnostic performance and making it better suited for clinical practice. Evolving OmiMRI with large-scale heterogeneous MRIs To assess the continual learning capacity of OmiMRI, we incrementally incorporated datasets from different centers into training, thereby generating progressively evolving model versions. OmiMRI processes arbitrary MRI combinations with minimal preprocessing requirements, facilitating direct utilization of multi-center clinical data and seamless model evolution. We curated a large-scale, multicenter brain MRI cohort covering six brain tumor and cerebrovascular diseases (Fig. 4A). Publicly available data were collected from 18 datasets on TCIA, BraTS2023, OpenNeuro, and ATLAS, totaling 30,938 3D MRIs. In-house data were obtained from 16 hospitals, with 60,843 3D MRIs in total. Public datasets showed substantial imbalance, with gliomas accounting for 74% of cases and a paucity of multiple disease types from the same center (Fig. 4A; Fig. S4A). In contrast, our in-house dataset is greatly balanced across six diseases, ranging from 11% for meningiomas to 23% for intracranial aneurysms (Fig. 4A; Fig. S4B). The superior diversity and balance of the in-house MRI dataset, together with its comprehensive disease coverage within single centers, provided a strong foundation for OmiMRI development. While multicenter datasets greatly expand scale and diversity, they also introduce pronounced heterogeneity. OmiMRI demonstrated strong capacity to accommodate such heterogeneity, thereby fully leveraging the advantages of large-scale MRI resources. In practice, this enables an iterative clinical development paradigm: initial prototypes (V1.0) can be rapidly built using data from a limited number of centers, followed by scaling, systematic optimization, and physician feedback to yield successive versions (V1.x) that continually evolve in real-world settings (Fig. 4B). For example, starting with HunanXiangya data to establish OmiMRI V1.0, we successively incorporated additional centers to produce OmiMRI V1.1, V1.2, and V1.3 (Fig. 4C). In two validation datasets (XiangyaChangde and ChenzhouRenmin4th), across two tasks (brain tumors vs. cerebrovascular diseases and glioma vs. meningioma), AUROC improved significantly with evolving version (Fig. 4D, E). Together, these results demonstrate OmiMRI’s ability to evolve through continual integration of heterogeneous MRI data, progressively enhancing its performance and clinical applicability. Practicality of OmiMRI in data-limited scenarios To evaluate the practicality of OmiMRI in data-limited scenarios, we conducted glioblastoma (GBM) and metastasis (MET) discrimination tasks, comparing its performance with that of neuroradiologists specializing in brain tumors. We used a small brain MRI dataset from HunanXiangya, comprising 546 subjects (GBM: 254; MET: 292) and 3,221 MRIs (GBM: 1,409; MET: 1,812) (Fig. 5A). During testing, the area under the receiver operating characteristic curve (AUROC) increased with the number of input MRIs across all three groups (Junior (n = 4), Senior (n = 3), and OmiMRI (n = 3)). With three MRIs, OmiMRI and the Senior group demonstrated comparable performance in AUROC (OmiMRI: 0.895, 95% CI: 0.878–0.906; Senior: 0.876, 95% CI: 0.843–0.904; P > 0.05) and the area under the precision-recall curve (AUPRC) (OmiMRI: 0.933, 95% CI: 0.922–0.939; Senior: 0.912, 95% CI: 0.880–0.936; P > 0.05). With four MRIs, OmiMRI achieved comparable AUROC (OmiMRI: 0.931, 95% CI: 0.909–0.948; Senior: 0.907, 95% CI: 0.873–0.927; P > 0.05) and significantly higher AUPRC than the Senior group (OmiMRI: 0.973, 95% CI: 0.964–0.979; (Senior: AUPRC: 0.930, 95% CI: 0.904–0.945; P < 0.05) (Fig. 5B–D; Fig. S5–S8). OmiMRI also clearly outperformed the Junior group when three or four combined MRIs were used in the inputs (Fig. 5B–D; Fig. S5–S8). In distinguishing GBM from MET, OmiMRI’s attention heatmaps closely corresponded with key radiological and pathological features. For GBM, OmiMRI exhibited broad, high-attention values throughout the entire tumor region, including the tumor core and surrounding edema, consistent with its infiltrative growth pattern and peripheral hypercellular margins (Fig. 5E). In contrast, for MET, the attention heatmap was more compact and confined to the tumor core, with well-defined boundaries that matched the pathological presentation of metastatic edema margins (Fig. 5F). Furthermore, across varying numbers, sequences, and imaging planes of input MRI combinations, OmiMRI consistently localized lesion areas within each MRI scan (Fig. 5E, F; Supplementary Fig. S9). OmiMRI achieves expert-level performance in discriminating GBM from MET using minimal training data. It identifies critical tumor features that align with established clinical knowledge across diverse MRI sequences and scan views. This concordance enhances interpretability and underscores its clinical relevance. These results demonstrate the strong performance and interpretability of OmiMRI, highlighting its practicality for clinical applications. Discussion Existing efforts to address clinical MRI variability can be broadly categorized into three paradigms: (i) harmonization-based approaches that attempt to normalize data distributions across sites 13,14 ; (ii) missing-modality approaches that assume a predefined modality set and explicitly impute absent inputs 27,28 ; and (iii) adaptive frameworks that treat heterogeneity itself as a modeling objective. OmiMRI belongs to the third category. Importantly, only this paradigm directly aligns with real-world clinical MRI acquisition, where input availability is inherently patient-specific and often unpredictable. OmiMRI presents a practical and unified AI framework for real-world clinical neuroimaging. It accepts arbitrary MRI inputs, enabling great practicality in complex hospital settings. By remaining agnostic to sequence type, count, and spatial properties, OmiMRI eliminates the notion of a canonical input space altogether. The system evolves with large-scale multi-center data, and matches or surpasses expert-level diagnosis in data-limited scenarios. Rather than treating variability in MRIs as a problem to be corrected 29,30 , we designed OmiMRI to inherently embrace this heterogeneity as a fundamental characteristic of clinical data. The framework operates on the principle that any combination of available MRI sequences (regardless of type, count, or spatial properties) should be seamlessly integrable for meaningful feature extraction and inference. By adopting this attribute-agnostic philosophy, OmiMRI eliminates the need for extensive data harmonization while preserving the rich, complementary information embedded across different MRI sequences. This approach effectively bridges the gap between the rigid requirements of conventional AI models and the dynamic, personalized nature of real-world clinical data. It is essential to distinguish OmiMRI from recent segmentation foundation models, such as MedSAM 31 and SAM-Med3D 32 . While these generalist models have achieved state-of-the-art (SOTA) performance on standardized benchmarks, they are primarily "format-rigid"—often restricted to one imaging input with specific channels (e.g., 3-channel RGB) or requiring manual prompting for optimal performance. In clinical practice, where multi-sequence availability is inherently inconsistent (e.g., missing T1c or T2f), these models may require sequence-specific retraining or data imputation. OmiMRI addresses this by serving as a "format-free" fusion framework. As demonstrated by our integration of various backbones (including SOTA brain MRI foundation models of BrainSegFounder and BrainIAC ), OmiMRI does not compete with these architectures but rather empowers them, unlocking their diagnostic potential in heterogeneous clinical environments without necessitating rigid input standardization. A key advantage of OmiMRI lies in its ability to accept arbitrary MRI inputs. Conventional AI models often require stringent data curation, leading to significant exclusion of clinically valuable scans 33,34 . In contrast, OmiMRI's attribute-agnostic design accommodates the inherent heterogeneity of clinical MRIs. This eliminates the need for extensive preprocessing and manual data harmonization, thereby reducing labor costs and minimizing biases introduced by subjective data selection. The framework makes full use of available clinical data, preserving informative variations often lost in standardized research datasets. The performance gains of OmiMRI stem from its capacity to integrate information from multiple MRI inputs 35,36 . Through self-attention–based fusion, the framework learns to identify and align complementary features across multiple modalities and imaging views, enabling more comprehensive characterization of pathological signatures than single-sequence models. Importantly, the attention patterns not only enhance performance but also provide clinically interpretable visualizations that correlate with established radiological knowledge. OmiMRI also possesses an intrinsic capability to handle missing modalities, which is a common challenge in clinical practice 27,28 . Unlike approaches that rely on explicit imputation techniques or require specialized architectures for different input combinations, OmiMRI dynamically adapts its feature extraction and fusion processes based on the available inputs. This flexibility offers a distinct practical advantage over fixed-input models, facilitating reliable inference even when certain sequences are unavailable. Beyond individual tasks, OmiMRI supports an incremental development paradigm that aligns with real-world research and clinical implementation timelines. Initial prototypes can be rapidly established using data from a limited number of centers. As more institutions contribute data, the framework systematically incorporates new heterogeneous information in the "format-free" manner, demonstrating progressive performance improvement across external validation sets. This process yields progressive performance improvements on external validation datasets without requiring complete retraining, making the framework particularly suitable for large-scale, multi-center collaboration. While OmiMRI demonstrates substantial clinical applicability, several promising directions for future development remain. First, self-supervised pre-training of 2D and 3D backbones on large-scale neuroimaging data could further improve feature learning and diagnostic accuracy. Second, while OmiMRI currently focuses on structural MRI (sMRI), integrating functional MRI (fMRI) would provide complementary dynamic brain information. The integration of both sMRI and fMRI data within the OmiMRI framework could significantly enhance the diagnosis of neuropsychiatric disorders. Third, the framework could be extended to incorporate additional clinical data types, such as textual diagnostic reports and pathological whole-slide images. Such multimodal integration would better reflect real-world clinical workflows and strengthen diagnostic performance across diverse medical scenarios. We view these directions as natural extensions of the framework rather than prerequisites for its current clinical utility. Materials and methods Curation of brain MRI dataset The Ethics Committee of Xiangya Hospital of Central South University (HunanXiangya) approved this retrospective analysis of anonymous patient data (No. 2025101736). HunanXiangya led and coordinated the entire data curation process. Data collection and analysis in this retrospective study were conducted anonymously. We retrospectively collected brain MRI data from 16 Chinese hospitals, covering six major brain diseases. The curated large-scale dataset spans brain tumors (BT) and cerebrovascular diseases (CVD), where BT includes glioma, meningioma, and metastasis, while CVD comprises infarction, aneurysm, and hemorrhage. The data collection period extended from 2005 to 2025. Following quality control and de-identification procedures, all data were pseudonymized with recoded identifiers before being included in the analysis. We selected 18 high-quality datasets focusing on the six brain diseases from over 100 publicly available sources 37 , including ten TCIA datasets, five BraTS2023 datasets, two OpenNeuro datasets, and ATLAS R2.0 stroke dataset. All brain MRIs underwent skull stripping to protect patient privacy and minimize background interference. Subsequently, the intensity values of each MRI were normalized to the 0-255 range based on its 5th and 99.9th percentile values. Analysis of brain MRI attributes Multi-parametric MRI sequences provide complementary information, revealing distinct structural and functional characteristics of the brain. Key sequences include: T1-weighted (T1w) imaging for anatomical detail; T2-weighted (T2w) for fluid and pathology detection; contrast-enhanced T1 (T1c) for assessing blood-brain barrier disruption; T2-fluidattenuated inversion recovery (FLAIR) (T2f) for suppressing cerebrospinal fluid signal; apparent diffusion coefficient (ADC) maps for quantifying water diffusion; diffusion-weighted imaging (DWI) for acute ischemia evaluation; susceptibility-weighted imaging (SWI) for detecting blood and calcifications; magnetic resonance angiography (MRA) for vascular visualization; and proton density (PD) weighting for tissue characterization. Spatially, MRIs are categorized as isotropic or anisotropic. Isotropic volumes are defined by a single spatial parameter of voxel resolution ( voxel_reso , mm/voxel), with values in this study predominantly falling within 0.75 < voxel_reso ≤ 1.0. For anisotropic acquisitions, four slice parameters were analyzed: slice resolution ( slice_reso , mm/pixel), categorized as ≤ 0.5, 0.5–1.0, or 1.0–2.0; slice step ( slice_step , mm), categorized as ≤ 0.5, 0.5–1.0, 1.0–2.0, 2.0–4.0, or > 4.0; slice number ( slice_num ), categorized as ≤ 20, 20–40, 40–80, or > 80; and slice view, categorized as axial, sagittal, or coronal. In clinical practice, the selection of MRI sequences and acquisition parameters is tailored to individual diagnostic needs, resulting in substantial variation in both the number and type of MRIs acquired per subject. Inter-subject consistency (i.e., the inverse of heterogeneity) was assessed for each dataset and attribute by first calculating the standard deviation (std) of MRI counts per subject. Consistency score was defined as exp(–std) , ranging from 0 to 1, with lower values indicating greater heterogeneity. For instance, in a research dataset where every subject undergoes identical isotropic T1w scanning, the std for both the T1w and voxel_reso attributes is zero, yielding a consistency score of 1. In contrast, clinical datasets exhibit lower consistency scores, reflecting their intrinsic heterogeneity arising from scanning individualization. Developing OmiMRI models Rationale and motivation. Several studies have attempted to address the complexity of clinical MRI data, such as aligning T1w MRI scans from different centers into a harmonized space 13,14 , and generating missing MRI sequences from acquired sequences—for instance, generating T2w MRI from T1w MRI 27,28 . However, site heterogeneity and modality missing represent only the tip of the iceberg when it comes to the complexity of heterogeneous MRI data in clinical settings. The development of attribute-agnostic AI models capable of handling extremely heterogeneous MRI data remains a critically important yet largely unexplored frontier. In the field of multimodal large language models (MM-LLM), a dominant and cutting-edge approach involves using multiple encoders to extract features from different modalities separately, then usually integrating these features using a Transformer-based model, enabling cross-modal interaction and fusion 38,39 . MM-LLM hold the capability to process inputs of varying lengths and quantities dynamically and adaptively. This functional adaptability characteristic bears resemblance to the “attribute-agnostic” nature inherent in clinical brain MRI practice, providing the foundational inspiration for the current study. However, due to the distinct characteristics of biomedical images, these existing MM-LLM models cannot be directly applied to MRI-based diagnosis. We conceptualize combined multi-sequence MRI inputs as a form of multimodal data representation. Within the OmiMRI framework, each individual MRI sequence undergoes independent feature extraction via general-purpose backbone encoders. The resulting features are then projected into a normalized embedding space to eliminate dimensional and semantic disparities. A transformer-based fusion module subsequently enables dynamic feature integration through self-attention mechanisms, effectively capturing cross-sequence dependencies. OmiMRI framework achieves intrinsic adaptability to clinical MRI heterogeneity without requiring predefined input specifications or modality alignment. Architecture. We began by systematically analyzing prevailing architectural paradigms for brain MRI analysis (Fig. S1). Current methodologies span a spectrum of input designs: from single 2D slice inputs where models process individual anatomical slices (Fig. S1A) 31,37 , to 3D volume analysis where entire MRI volumes are processed holistically (Fig. S1B) 32,37 . For multi-sequence (T1w, T2w, T1c, and T2f) integration, two predominant strategies have emerged: 4D concatenation approaches where co-registered isotropic sequences are stacked along a channel dimension (Fig. S1C) 40,41 ; and modality-specific encoder designs where dedicated pathways process predefined sequences followed by feature fusion (Fig. S1D) 42–44 . These multi-sequence methods share fundamental constraints—they require isotropic, co-registered inputs and predefined modality combinations. These requirements fundamentally limit their applicability to real-world clinical data, where heterogeneity in acquisition protocols and frequent modality missing are inherent characteristics. In response, we developed OmiMRI as a unified, attribute-agnostic framework that accepts arbitrary MRI combinations (Fig. 1, Fig. S2). The architecture employs general backbone networks to independently extract features for all MRIs in the combined input, which are linearly projected into a unified embedding space to normalize dimensional and semantic disparities. A transformer-based fusion module then enables cross-sequence feature interaction through multi-head self-attention mechanisms, effectively capturing long-range dependencies across variable-input contexts. This design eliminates requirements for isotropic resolution, spatial co-registration, or predefined modality sets, thereby directly processing native clinical MRI data without structural harmonization. The framework's modular organization further ensures extensibility to emerging network architectures and additional neuroimaging modalities. Inputs. A fundamental capability of OmiMRI is processing combined inputs with variable MRI counts, where the set size S represents the number of MRIs per input. To construct these combined inputs, for a subject with N available MRI scans, the number of possible distinct subsets of size S is given by the binomial coefficient C(N, S) . We evaluated S values of 1, 2, 3, and 4. To ensure dataset balance across different set sizes, when C(N, S) exceeded 2× N , we randomly sampled a maximum of 2× N combinations. Within each combined input, the S MRIs were presented in a randomized order at every iteration to prevent the model from learning any spurious correlations based on the sequence order. This approach of combined input construction directly addresses the heterogeneity in the number of MRIs per patient encountered in clinical practice. Preprocessing. To enable batch processing in the volume MRI experiments, input data were formatted into a 6-dimensional tensor with layout BSDCHW . Here, B denotes batch size; S represents set size; D indicates depth, corresponding to the count of effective slices per MRI (with effective slices defined as containing >10% non-zero pixels). The depth dimension was standardized to D through either resampling or zero-padding. In preliminary experiments, we evaluated D values of 24, 32, 48, and 64, ultimately selecting D =32 for all subsequent experiments as it provided the optimal balance between performance and computational efficiency. C represents channels (fixed at C =3 following conversion of grayscale MRI slices to RGB format to leverage pre-trained weights from natural image/video models). H and W denote slice height and width, standardized through resizing. Height H and width W were standardized through resizing. Preliminary tests with shapes of 128×128, 192×192, and 256×256 led to the optimal selection of 192×192 for classification and regression and 256×256 for segmentation. This spatial shape preprocessing pipeline standardizes the DHW dimensions while remaining agnostic to MRI sequence type or scanning orientation. For 2D MRI slice experiments, depth dimension D =1 . Group batching. In practical model development, batch processing is essential for achieving efficient and effective training. We implemented a group batching strategy stratified by the set size S . Fist, organize the datasets into distinct groups according to their S values, with each group containing samples of identical set sizes. Then, given a predefined batch size B , we randomly select B samples from the same group to form a 6-dimensional input following the BSDCHW layout. During training, each iteration randomly loads batches from different set-size groups. This group batching approach effectively handles data with varying set sizes while maintaining computational efficiency during deep model training and inference. Training. OmiMRI integrates dynamically swappable backbones and task-specific heads to achieve substantial versatility. The dimensions of the raw features ( Dx and Lx ) depend on the backbone architecture and input data shape. Following feature normalization, the feature dimension is standardized to D0 = 128, with a channel count of Ly = 10. During feature concatenation, the S feature maps are stacked along the channel dimension. After self-attention computation, a 512-dimensional feature vector is obtained, which is subsequently passed through a linear layer for final classification or regression. In preliminary 3D experiments, we evaluated various backbones with pretrained parameters from the MMAction2 platform, including ResNet50_SlowOnly_TPN , ResNet50_SlowFast 45 , ResNet34_R2plus1D 46 , ViT_Video_small 47 , and SwinT_Video_small 48 . ResNet50_SlowOnly_TPN was selected as the best backbone for OmiMRI, offering the optimal balance between performance and efficiency. For 2D experiments, we tested backbones with pretrained parameters from Torchvision, including ResNet18, ResNet50, MobileNet v2 , and ViT . ResNet50 was ultimately chosen as the best backbone for OmiMRI based on its balanced performance. In segmentation tasks, we evaluated DeepLab v3 networks 49,50 with ResNet18, ResNet50, HRNet v2 , and MobileNet v2 backbones during preliminary experiments. The DeepLab v3 architecture with a ResNet50 backbone was selected for OmiMRI's segmentation tasks. These selections were guided by systematic evaluation of each architecture's compatibility with our heterogeneous MRI data characteristics. ResNet50 variants consistently demonstrated superior adaptability to the varied spatial resolutions and sequence parameters in our clinical datasets, while maintaining computational feasibility for hospital deployment. Our current implementation represents an initial configuration within OmiMRI's flexible architecture. The framework's modular design intentionally accommodates future integration of emerging backbone architectures as they become available. We anticipate that with substantially expanded dataset scales, transformer-based architectures such as ViT and SwinT variants may demonstrate advantages in capturing complex patterns across heterogeneous data. This forward compatibility ensures continuous performance improvement while maintaining consistency in clinical deployment, allowing the framework to evolve with advancing model architectures and growing data resources. All experiments were conducted on a Biomedical Computing Platform utilizing NVIDIA A800 GPUs with 80GB memory. We employed consistent general parameter settings across all tasks: AdamW optimizer with a base learning rate of 0.0001, L2 weight decay of 0.0005, and early stopping patience of 6 epochs. For loss functions, we used CrossEntropyLoss for classification tasks, SmoothL1Loss for regression tasks, and FocalLoss for segmentation tasks to address class imbalance. Task-specific parameters were optimized for each experimental paradigm. For 3D volumetric analysis, we used a batch size of 16 with 2 gradient accumulation steps, training for 1,000-10,000 steps with validation every 200 steps. For 2D classification tasks, we set a batch size of 32 with 2 gradient accumulation steps, training for 200-5,000 steps with validation every 20 steps. For 2D segmentation tasks, we maintained a batch size of 32 but increased gradient accumulation to 4 steps, training for 2,000-20,000 steps with validation every 500 steps to accommodate the increased computational demands of pixel-wise prediction. This stratified parameter configuration achieves an optimal balance between model performance and computational efficiency across different task types and data modalities. Tasks and experiments 3D Volume Analysis. For volume classification, multiple public datasets were utilized. The glioma versus metastasis (GLI vs. MET) and glioma versus meningioma (GLI vs. MEN) tasks were constructed using the BraTS2023 dataset. Low-grade glioma versus high-grade glioma (LGG vs. HGG) classification was performed with the BraTS2020 dataset. The LGG versus glioblastoma (GBM) task incorporated both TCGA-LGG and TCGA-GBM datasets, while isocitrate dehydrogenase (IDH) mutation prediction was developed using TCGA-LGG and UPENN-GBM datasets. For the volume regression task, brain age prediction was implemented with the IXI dataset. For all tasks, the dataset was split into training, validation, and test sets at the subject level using a 7:1:2 ratio. To establish performance baselines, we compared OmiMRI against several established 3D architectures based on MMAction2 platform, including ResNet50_SlowOnly_TPN (Conv3D TPN), ResNet34_R2plus1D (Conv2D+1D), and ResNet50_SlowFast (Conv3D SlowFast) models. These conventional approaches typically process single MRI sequences and require standardized input formats, serving as representative benchmarks for current volume-based analysis methods. 2D Image Analysis. For 2D image classification, slice-level datasets were constructed from BraTS2023 and BraTS2020 volumetric MRI data. Effective slices were extracted along the axial plane, with tumor-positive slices defined as containing tumor regions exceeding 20 pixels, and tumor-negative slices containing no tumor pixels. To maintain dataset balance, all positive slices from each subject were retained, while negative slices were randomly subsampled at 25%. The glioma recognition (GLI Recog) task distinguished glioma-positive from glioma-negative slices; meningioma recognition (MEN Recog) differentiated meningioma-positive from glioma-negative slices; and LGG vs. HGG classification was performed using tumor-positive slices from BraTS2020. For 2D segmentation tasks, we constructed slice-level datasets from the BraTS2023 glioma and meningioma collections. Segmentation datasets were annotated with three distinct tumor subregions: non-enhancing tumor core (NETC), surrounding non-enhancing FLAIR hyperintensity (SNFH), and enhancing tumor (ET). For all tasks, the dataset was split into training, validation, and test sets at the subject level using a 7:1:2 ratio. For comparative evaluation in 2D tasks, we implemented several widely-used architectures. In classification tasks, ResNet50 , MobileNet v2 , and ResNet18 based on Torchvision served as baseline models, representing current standard approaches for single-image analysis. For segmentation tasks, we benchmarked against DeepLab v3 with ResNet50 , HRNet v2 , and MobileNet v2 backbones, which constitute representative contemporary methods for medical image segmentation. OmiMRI vs. fixed-input models: Most of existing models requires fixed -input. Six different backbone architectures were tested, among which CT-FM 51 , BrainSegFounder 52 , BrainIAC 53 , and BrainLaMIM 41 are pretrained medical imaging foundation models, while Conv2D+1D and Conv3D-TPN are models pretrained on natural videos. Each Fixed model can only process a fixed number of inputs; thus, four separate fixed models were trained for scenarios with 1, 2, 3, and 4 input MRIs respectively. In contrast, the OmiMRI framework requires training only a single model to perform inference with any arbitrary number of inputs. We evaluated the models on two 3D classification tasks: glioma grading (LGG versus HGG) and IDH mutation prediction. OmiMRI evolving experiment. In the evolving experiments, large-scale multi-center data were grouped by institution and sequentially introduced into the training set to produce progressively evolved model versions. The substantial dataset from HunanXiangya served as the foundation, yielding the initial model version V1.0. Subsequent versions were developed by incrementally expanding the training data: V1.1 incorporated data from HubeiTCM, YichangZhongxin, YueyangLinxiangRenmin, and ChenzhouRenmin3rd; V1.2 added data from HunanUnivMedicine, HunanRenmin, HuaihuaTCM, and ZhuzhouChalingRenmin; and V1.3 further integrated data from YueyangZhongxin, YunnanRenmin1st, Changsha4th, and Zhuzhou331. All data from these centers were divided into training, validation, and test sets at the subject level using a 7:1:2 ratio. External validation was performed using data from XiangyaChangde and ChenzhouRenmin4th, where 10% of the data supported model fine-tuning, 70% was used for metric evaluation, and 20% was held in reserve. The evolving experiments evaluated OmiMRI on two clinical discrimination tasks: brain tumor versus cerebrovascular disease, and glioma versus meningioma. Data-limited GBM vs. MET. Glioblastoma (GBM) and brain metastasis (MET) represent two highly malignant brain tumors with overlapping radiological presentations but distinct pathogenesis and treatment regimens. Accurate discrimination between them carries significant implications for surgical planning and therapeutic strategy. We curated a clinically challenging, small-scale brain MRI dataset from HunanXiangya Hospital, comprising patients diagnosed between 2010 and 2025. Following rigorous manual quality control, the final cohort included 546 subjects and 3,221 MRIs. For OmiMRI development, we constructed combination samples containing one to four MRIs per input. These samples were divided into training, validation, and test sets using a subject-level split ratio of 7:1:2. Saliency heatmaps for the test dataset were generated using the Grad-CAM method, providing visual explanations of the model's focus areas during decision-making. In the human expert evaluation, seven neuroradiologists (four junior and three seniors, with clinical experience ranging from 3 to 22 years) specializing in brain tumors independently diagnosed 727 randomly selected samples from the test set. The expert evaluation dataset consisted of 200 samples with a single MRI, 200 samples with 2–3 MRIs per combined input, and 127 samples with four MRIs per combined input. Each expert assessed the probability of GBM and MET for every sample using a discrete scoring system with increments of 0.1, with the constraint that the sum of the probability scores for GBM and MET must equal 1.0. Evaluation. The area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC) were employed for classification tasks. Mean absolute error (MAE) was used for the brain age prediction regression task. The Dice score was adopted for segmentation tasks. For each task, the OmiMRI model was independently trained three times, and the average evaluation metrics were calculated. Declarations Data & Code Availability Public datasets used in this study are available from their original sources. Aggregated statistics, cohort summaries, and selected derived data supporting the findings are provided in the Supplementary Information; additional data related to this study are being further curated for consolidated reference and may be made available for academic research upon reasonable request and subject to institutional approval. The implementation of the OmiMRI framework, including model architectures, training scripts, and evaluation pipelines, will be made publicly available upon publication via a permanent online repository (e.g., GitHub or Zenodo). Pretrained model weights and configuration files corresponding to the experiments reported in this study will also be released to facilitate reproducibility and future benchmarking. Acknowledgements We thank the Biomedical Computing Platform of National Biomedical Imaging Center at Peking University for providing computational resources. We express our gratitude to the neuroradiologists who participated in the glioblastoma (GBM) and metastasis (MET) discrimination tasks. This work is supported by Beijing Natural Science Foundation (7254542, JQ24023, F251020), the Beijing Municipal Science and Technology Commission Project (Grant No. Z231100006623010), China Postdoctoral Science Foundation (8206301107), and the Peking University Medicine Plus X Pilot Program – Artificial Intelligence and Medical Development Initiative (BMU2025YXXLHAIYX020). References Eisenstein, M. Pushing the limits of MRI brain imaging. Nat Methods 21 , 1975–1979 (2024). Hess, C. P. MRI of the Brain: What Is Driving Innovation in 2023? Radiology 308 , e231657 (2023). Lohrke, J. et al. 25 Years of Contrast-Enhanced MRI: Developments, Current Challenges and Future Perspectives. Adv Ther 33 , 1–28 (2016). Ali, S. S. A., Memon, K., Yahya, N. & Khan, S. Deep learning frameworks for MRI-based diagnosis of neurological disorders: a systematic review and meta-analysis. Artif Intell Rev 58 , 171 (2025). Parizel, P. M. et al. Magnetic Resonance Imaging of the Brain. in Clinical MR Imaging (eds Reimer, P., Parizel, P. M., Meaney, J. F. M. & Stichnoth, F. A.) 107–195 (Springer Berlin Heidelberg, Berlin, Heidelberg, 2010). doi:10.1007/978-3-540-74504-4_2. Yang, Z. et al. Brain aging patterns in a large and diverse cohort of 49,482 individuals. Nat Med 30 , 3015–3026 (2024). Schaff, L. R. & Mellinghoff, I. K. Glioblastoma and Other Primary Brain Malignancies in Adults: A Review. JAMA 329 , 574–587 (2023). Tanzhu, G. et al. Metastatic brain tumors: from development to cutting‐edge treatment. MedComm 6 , e70020 (2025). Zhang, Y. et al. Glioblastoma-associated macrophages in glioblastoma: from their function and mechanism to therapeutic advances. Cancer Gene Ther 32 , 595–607 (2025). Reyna, M. A., Nsoesie, E. O. & Clifford, G. D. Rethinking Algorithm Performance Metrics for Artificial Intelligence in Diagnostic Medicine. JAMA 328 , 329–330 (2022). Rosenthal, J. T., Beecy, A. & Sabuncu, M. R. Rethinking clinical trials for medical AI with dynamic deployments of adaptive systems. npj Digit. Med. 8 , 252 (2025). Zhang, A., Xing, L., Zou, J. & Wu, J. C. Shifting machine learning for healthcare from development to deployment and from models to data. Nat. Biomed. Eng 6 , 1330–1345 (2022). Cackowski, S., Barbier, E. L., Dojat, M. & Christen, T. ImUnity: A generalizable VAE-GAN solution for multicenter MR image harmonization. Med Image Anal 88 , 102799 (2023). Roca, V. et al. IGUANe: A 3D generalizable CycleGAN for multicenter harmonization of brain MR images. Med Image Anal 99 , 103388 (2025). Martella, M., Lenzi, J. & Gianino, M. M. Diagnostic Technology: Trends of Use and Availability in a 10-Year Period (2011–2020) among Sixteen OECD Countries. Healthcare (Basel) 11 , 2078 (2023). Smith-Bindman, R. et al. Trends in Use of Medical Imaging in US Health Care Systems and in Ontario, Canada, 2000-2016. JAMA 322 , 843–856 (2019). Pär, K. Number of MRI Scans Per Year Worldwide: Overview of global MRI utilization. https://collectiveminds.health/ (2024). Clark, K. et al. The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. J Digit Imaging 26 , 1045–1057 (2013). Kennedy, D. N., Haselgrove, C., Riehl, J., Preuss, N. & Buccigrossi, R. The NITRC image repository. NeuroImage 124 , 1069–1073 (2016). Sudlow, C. et al. UK Biobank: An Open Access Resource for Identifying the Causes of a Wide Range of Complex Diseases of Middle and Old Age. PLoS Med 12 , e1001779 (2015). Markiewicz, C. J. et al. The OpenNeuro resource for sharing of neuroscience data. eLife 10 , e71774 (2021). Sun, Y., Wang, L., Li, G., Lin, W. & Wang, L. A foundation model for enhancing magnetic resonance images and downstream segmentation, registration and diagnostic tasks. Nat. Biomed. Eng 1–18 (2024) doi:10.1038/s41551-024-01283-7. Wang, L. et al. A lifespan-generalizable skull-stripping model for magnetic resonance images that leverages prior knowledge from brain atlases. Nat. Biomed. Eng 1–16 (2025) doi:10.1038/s41551-024-01337-w. Elyounssi, S. et al. Addressing artifactual bias in large, automated MRI analyses of brain development. Nat Neurosci 28 , 1787–1796 (2025). Moor, M. et al. Foundation models for generalist medical artificial intelligence. Nature 616 , 259–265 (2023). He, J. et al. The practical implementation of artificial intelligence technologies in medicine. Nature Medicine 25 , 30–36 (2019). Hu, X. B. et al. AutoGAN-Synthesizer: Neural Architecture Search for Cross-Modality MRI Synthesis. Medical Image Computing and Computer Assisted Intervention (MICCAI 2022) 13436 , 397–409 (2022). Li, H. W. et al. DiamondGAN: Unified Multi-modal Generative Adversarial Networks for MRI Sequences Synthesis. Medical Image Computing and Computer Assisted Intervention (MICCAI 2019) 11767 , 795–803 (2019). Orlhac, F. et al. How can we combat multicenter variability in MR radiomics? Validation of a correction procedure. Eur Radiol 31 , 2272–2280 (2021). Da-Ano, R., Visvikis, D. & Hatt, M. Harmonization strategies for multicenter radiomics investigations. Phys. Med. Biol. 65 , 24TR02 (2020). Ma, J. et al. Segment anything in medical images. Nat Commun 15 , 654 (2024). Wang, H. et al. SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images. Preprint at https://doi.org/10.48550/arXiv.2310.15161 (2024). Hagler, D. J. et al. Image processing and analysis methods for the Adolescent Brain Cognitive Development Study. NeuroImage 202 , 116091 (2019). Alfaro-Almagro, F. et al. Image processing and Quality Control for the first 10,000 brain imaging datasets from UK Biobank. NeuroImage 166 , 400–424 (2018). Zhao, H., Cai, H. & Liu, M. Transformer based multi-modal MRI fusion for prediction of post-menstrual age and neonatal brain development analysis. Medical Image Analysis 94 , 103140 (2024). Liu, Z., Wei, J., Li, R. & Zhou, J. Learning multi-modal brain tumor segmentation from privileged semi-paired MRI images with curriculum disentanglement learning. Computers in Biology and Medicine 159 , 106927 (2023). Su, F. et al. From Slices to Volumes: A Scalable Pipeline for Developing General-Purpose Brain MRI Foundation Models. Preprint at https://doi.org/10.1101/2025.04.12.25325728 (2025). Yin, S. et al. A Survey on Multimodal Large Language Models. National Science Review 11 , nwae403 (2024). Zhang, D. et al. MM-LLMs: Recent Advances in MultiModal Large Language Models. in Findings of the Association for Computational Linguistics ACL 2024 12401–12430 (Association for Computational Linguistics, Bangkok, Thailand and virtual meeting, 2024). doi:10.18653/v1/2024.findings-acl.738. Liu, H. et al. M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities. Preprint at http://arxiv.org/abs/2303.05302 (2023). Chen, M. et al. Medical image foundation models in assisting diagnosis of brain tumors: a pilot study. Eur Radiol https://doi.org/10.1007/s00330-024-10728-1 (2024) doi:10.1007/s00330-024-10728-1. Shi, J. et al. PASSION: Towards Effective Incomplete Multi-Modal Medical Image Segmentation with Imbalanced Missing Rates. Preprint at https://doi.org/10.48550/arXiv.2407.14796 (2024). Pipoli, V. et al. IM-Fuse: A Mamba-Based Fusion Block for Brain Tumor Segmentation with Incomplete Modalities. in Medical Image Computing and Computer Assisted Intervention – MICCAI 2025 (eds Gee, J. C. et al.) 225–235 (Springer Nature Switzerland, Cham, 2026). Zeng, Z., Peng, Z., Yang, X. & Shen, W. Missing as Masking: Arbitrary Cross-Modal Feature Reconstruction for Incomplete Multimodal Brain Tumor Segmentation. in Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 (eds Linguraru, M. G. et al.) 424–433 (Springer Nature Switzerland, Cham, 2024). Feichtenhofer, C., Fan, H., Malik, J. & He, K. SlowFast Networks for Video Recognition. in 2019 IEEE/CVF International Conference on Computer Vision (ICCV) 6201–6210 (2019). doi:10.1109/ICCV.2019.00630. Tran, D. et al. A Closer Look at Spatiotemporal Convolutions for Action Recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition 6450–6459 (2017). Tong, Z., Song, Y., Wang, J. & Wang, L. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. in Advances in Neural Information Processing Systems (eds Koyejo, S. et al.) vol. 35 10078–10093 (Curran Associates, Inc., 2022). Liu, Z. et al. Video Swin Transformer. in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 3192–3201 (2022). doi:10.1109/CVPR52688.2022.00320. Chen, L.-C., Papandreou, G., Schroff, F. & Adam, H. Rethinking Atrous Convolution for Semantic Image Segmentation. Preprint at https://doi.org/10.48550/arXiv.1706.05587 (2017). Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F. & Adam, H. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. Preprint at https://doi.org/10.48550/arXiv.1802.02611 (2018). Pai, S. et al. Vision Foundation Models for Computed Tomography. Preprint at https://doi.org/10.48550/arXiv.2501.09001 (2025). Cox, J. et al. BrainSegFounder: Towards 3D foundation models for neuroimage segmentation. Medical Image Analysis 97 , 103301 (2024). Tak, D. et al. A foundation model for generalized brain MRI analysis. 2024.12.02.24317992 Preprint at https://doi.org/10.1101/2024.12.02.24317992 (2024). Table 1 Table 1. Comparison of F1-scores between OmiMRI models and Fixed models Model name (backbone) Model type Glioma grading (LGG vs. HGG) IDH mutation prediction Count of input MRIs Count of input MRIs 1 2 3 4 1 2 3 4 CT-FM (ResNet) Fixed 0.768 0.778 0.783 0.772 0.559 0.717 0.804 0.834 Omi 0.779 0.804 0.814 0.830 0.556 0.727 0.805 0.841 BrainSegFounder (SwinT) Fixed 0.740 0.748 0.746 0.751 0.569 0.741 0.809 0.829 Omi 0.777 0.825 0.839 0.846 0.556 0.731 0.818 0.848 BrainIAC (ViT) Fixed 0.766 0.748 0.736 0.788 0.572 0.676 0.731 0.780 Omi 0.733 0.783 0.791 0.791 0.598 0.746 0.809 0.826 BrainLaMIM (ViT) Fixed 0.747 0.795 0.769 0.779 0.503 0.652 0.714 0.711 Omi 0.776 0.817 0.824 0.827 0.551 0.692 0.775 0.800 Conv2D+1D (ResNet34) Fixed 0.810 0.828 0.818 0.811 0.693 0.769 0.832 0.872 Omi 0.817 0.833 0.834 0.836 0.640 0.768 0.847 0.875 Conv3D-TPN (ResNet50) Fixed 0.814 0.783 0.845 0.869 0.674 0.775 0.827 0.845 Omi 0.843 0.885 0.913 0.919 0.639 0.784 0.851 0.874 Six different backbone architectures were tested. Bold text indicates instances where the OmiMRI model achieved superior performance compared to the Fixed model with the same backbone and input. For each task, both the OmiMRI models and Fixed models utilized identical training, validation, and test datasets. For each model, the parameter counts of the encoder, Omi-Neck & Head, and Fixed-Neck & Head are presented in Table S1. Note that for the 'Fixed' models, four separate models had to be trained and deployed for each input combination, whereas OmiMRI used a single model for all scenarios. Additional Declarations There is NO Competing Interest. Supplementary Files SupplementaryMaterials.docx Cite Share Download PDF Status: Under Review Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8651933","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":577637371,"identity":"8837bcf0-d428-4a33-b16f-6fd798f76b00","order_by":0,"name":"Lei Ma","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAxklEQVRIiWNgGAWjYBACPgbGBiBlwyAB4vEQo4UNoiWNJC1gcJgULezNDUw3as7nSc5IYHzwto1B3pygFp6DDcw5x24XS0skMBvObWMw3NlASItEIlAL2+3EeRIJbNK8bQwJBgcIaZF/CNTy7xxIC/tv4rRIMDYw57YdSJwNtIWZOC08iQ2Hc/uSiyV7HjZLzjknYbiBkBZ+9uMPH+d8s8uTOJ588MObMht5graAAEhNAgMkTiWIUA8FCcQrHQWjYBSMghEHACEtOka6PdBxAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0001-6024-3854","institution":"Peking Univeristy","correspondingAuthor":true,"prefix":"","firstName":"Lei","middleName":"","lastName":"Ma","suffix":""},{"id":577637372,"identity":"a261f5dc-fc7a-48be-b97d-4169d7e0e37b","order_by":1,"name":"Feng Su","email":"","orcid":"","institution":"Peking Univeristy","correspondingAuthor":false,"prefix":"","firstName":"Feng","middleName":"","lastName":"Su","suffix":""},{"id":577637373,"identity":"c629fa39-ffd4-4db8-8618-0e9859550aa8","order_by":2,"name":"Xiaoping Yi","email":"","orcid":"","institution":"Chongqing University","correspondingAuthor":false,"prefix":"","firstName":"Xiaoping","middleName":"","lastName":"Yi","suffix":""},{"id":577637374,"identity":"5c859bcd-6106-4be3-b109-65ced08617f7","order_by":3,"name":"Ye Cheng","email":"","orcid":"","institution":"Capital Medical University","correspondingAuthor":false,"prefix":"","firstName":"Ye","middleName":"","lastName":"Cheng","suffix":""},{"id":577637375,"identity":"501512e2-b36b-41a2-934e-afa6c24a22a7","order_by":4,"name":"Yongjie Ma","email":"","orcid":"","institution":"China International Neuroscience Institute, Xuanwu Hospital, Capital Medical University","correspondingAuthor":false,"prefix":"","firstName":"Yongjie","middleName":"","lastName":"Ma","suffix":""},{"id":577637376,"identity":"af0838e3-64c2-45e4-8777-ac1a33bb8a0a","order_by":5,"name":"Zeming Tan","email":"","orcid":"","institution":"Central South University","correspondingAuthor":false,"prefix":"","firstName":"Zeming","middleName":"","lastName":"Tan","suffix":""},{"id":577637377,"identity":"008710be-89d2-49a2-8e7a-35bd63425c75","order_by":6,"name":"Gengdi Huang","email":"","orcid":"","institution":"Southern University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Gengdi","middleName":"","lastName":"Huang","suffix":""}],"badges":[],"createdAt":"2026-01-20 18:11:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8651933/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8651933/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":100761215,"identity":"27bf3c77-1ca3-4d1c-87cf-eb010d8841a6","added_by":"auto","created_at":"2026-01-21 07:37:29","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":1903085,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eArchitecture of OmiMRI: a unified and practical AI framework for heterogeneous clinical brain MRIs.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eA. Representative examples of the diverse MRI modalities and scanning attributes typically encountered in clinical practice, highlighting the extensive variation in visual attributes.\u003c/p\u003e\n\u003cp\u003eB. Challenge and solution schematics. The top panel delineates the sources of immense heterogeneity in brain MRI data, ranging from acquisition parameters and pathological conditions to inter-center variability. The bottom panel contrasts the conventional format-rigid modeling approach—which forces data exclusion and waste due to strict input standardization (left)—with the OmiMRI format-free modeling paradigm that accommodates arbitrary, heterogeneous inputs (right).\u003c/p\u003e\n\u003cp\u003eC. The OmiMRI workflow and network architecture. The upper section illustrates the efficiency of OmiMRI: unlike conventional methods that require distinct models (Models α–γ) for every unique combination of input modalities, OmiMRI uses a single unified model to handle varying inputs, significantly reducing preprocessing labor while improving data utility. The lower section details the universal architecture, which accommodates a series of fixed-input, pretrained backbones (for example, BrainSegFounder, BrainIAC, or 2D/3D CNNs) as feature encoders. Extracted features are aligned via linear projection and normalization. The framework then employs task-specific fusion strategies to establish a clinically adaptive system, including concatenation with self-attention for linear head tasks (classification and regression) or dynamic weighting for dense prediction (segmentation).\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-8651933/v1/7cc7c86125d6443f8a08d92a.png"},{"id":100761202,"identity":"b2031a36-f89b-46b1-a173-ee7a56ca712a","added_by":"auto","created_at":"2026-01-21 07:37:24","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":3733274,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eInherent heterogeneity in clinical brain MRI.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eA. Bubble plot summarizing brain MRI consistency across multiple attributes. The horizontal axis represents 27 distinct datasets, with 16 in-house datasets (black fonts), and 11 open-source datasets (blue fonts). The vertical axis lists 26 MRI attributes. Each bubble reflects the statistical distribution of MRI counts for a specific attribute across all subjects in a dataset. Bubble size corresponds to the average count of MRIs, and color indicates inter-subject consistency (i.e., the inverse of heterogeneity). Darker colors represent higher consistency (lower heterogeneity) in MRI attributes within a dataset. A vertical dashed line divides the data into research-oriented and practical clinical datasets. The five leftmost datasets to the left of the dashed line are research-oriented, characterized by simple attributes and high inter-subject consistency. In contrast, practical clinical datasets to the right of the dashed line exhibit highly complex attributes and substantially lower inter-subject consistency.\u003c/p\u003e\n\u003cp\u003eB. Sankey diagram illustrating T1w MRI variability in the RocklandSample, ABIDE-1, and HunanXiangya datasets. Four scan slice parameters (slice view, resolution (reso), number (num), and step) are analyzed. Diagram complexity reflects the degree of inter-subject MRI heterogeneity. All samples in the RocklandSample dataset demonstrate perfectly consistent slice parameters. The increased complexity of flow lines highlights the pronounced heterogeneity present in clinical MRI data, especially in HunanXiangya dataset.\u003c/p\u003e\n\u003cp\u003eC. Sankey diagram of T2w MRI for the YunnanRenmin1st and HunanXiangya datasets.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-8651933/v1/df142e87e7cd0f7bc65d65fd.png"},{"id":100761283,"identity":"f8b11a48-fc88-4f65-bf38-e727c6ae54d6","added_by":"auto","created_at":"2026-01-21 07:39:19","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":1823087,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003ePerformance of OmiMRI across classification, regression, and segmentation tasks.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe OmiMRI framework can handle arbitrary input combinations and achieve effective feature fusion, whereas conventional models are typically restricted to a fixed input format. OmiMRI_I1, OmiMRI_I2, OmiMRI_I3, OmiMRI_I4, and OmiMRI_I3+ denote the same model performing inference using combined inputs with 1, 2, 3, 4, and ≥3 MRIs, respectively. Abbreviations: glioma (GLI), metastasis (MET), meningioma (MEN), low-grade glioma (LGG), high-grade glioma (HGG), glioblastoma (GBM), isocitrate dehydrogenase (IDH), non-enhancing tumor core (NETC), surrounding non-enhancing FLAIR hyperintensity (SNFH), enhancing tumor (ET), and area under receiver operating characteristic (ROC) curve (AUROC).\u003c/p\u003e\n\u003cp\u003eA. Progressive performance improvement in LGG versus HGG classification with multi-modal MRIs. Example MRIs from an HGG subject are shown. OmiMRI_I1 through OmiMRI_I4 represent the same model utilizing one to four MRIs for inference. Sequential addition of T1w, T2w, T1c, and T2f sequences progressively increases the predicted probability of HGG.\u003c/p\u003e\n\u003cp\u003eB. Summary plot of AUROC values for LGG versus HGG classification. The horizontal axis indicates sequential incorporation of T1w, T2w, T1c, and T2f sequences. Left panel: volume classification tasks. Right panel: image classification tasks.\u003c/p\u003e\n\u003cp\u003eC. Progressive performance improvement in tumor segmentation with multi-modal MRIs. Example segmentation results demonstrate improved tumor subregion delineation as additional sequences are incorporated.\u003c/p\u003e\n\u003cp\u003eD. Summary plot of mean Dice scores for tumor segmentation tasks. The horizontal axis represents sequential integration of T1w, T2w, T1c, and T2f sequences, showing progressive improvement in segmentation performance.\u003c/p\u003e\n\u003cp\u003eE. Performance metrics for volume classification and regression tasks. The first column indicates model names; subsequent columns display AUROC across five classification tasks and mean absolute error (MAE) for brain age prediction task. Comparative performance is illustrated for OmiMRI_I3+, OmiMRI_I2, OmiMRI_I1, Conv3D TPN, Conv2D+1D, and Conv3D SlowFast models.\u003c/p\u003e\n\u003cp\u003eF. Dice scores for image segmentation tasks. Comparative performance is illustrated for OmiMRI_I3+, OmiMRI_I2, OmiMRI_I1, Deeplab v3 with ResNet50, HRNet v2, and MobileNet v2.\u003c/p\u003e\n\u003cp\u003eG. AUROC for image classification tasks. Comparative performance is illustrated for OmiMRI_I3+, OmiMRI_I2, OmiMRI_I1, ResNet50, MobileNet v2, and ResNet18.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-8651933/v1/97924a3ae04d1db6cda1c0d6.png"},{"id":100761282,"identity":"c353979c-d4ae-4a3b-8e8b-13889357ea31","added_by":"auto","created_at":"2026-01-21 07:39:18","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":2747099,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eEvolving OmiMRI through continuous integration of large-scale multi-center data.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eA. Curation of large-scale brain MRIs spanning brain tumors (BT) and cerebrovascular diseases (CVD). BT includes glioma, meningioma, and metastasis; CVD includes infarction, aneurysm, and hemorrhage. The vertical axis lists four public MRI collections (blue font) and 16 in-house datasets (black font). \"TCIA (10)\" denotes ten TCIA datasets; \"BraTS2023 (5)\" indicates five BraTS2023 datasets; \"OpenNeuro (2)\" represents two OpenNeuro datasets. Left panel: summary of MRI counts. Right panel: summary of patient counts. Inset panels: proportional distribution of in-house versus public brain MRIs across the six disease types.\u003c/p\u003e\n\u003cp\u003eB. Pipeline for iterative OmiMRI development. An initial prototype (version V1.0) is rapidly established using heterogeneous brain MRIs from selected centers. The model is subsequently refined (versions V1.x) by incorporating additional multi-center datasets and incorporating expert interactions.\u003c/p\u003e\n\u003cp\u003eC. Example implementation of the evolving OmiMRI framework. Training data were partitioned by institution into four groups and sequentially introduced during training, producing evolving model versions (V1.0 to V1.3). External validation datasets come from XiangyaChangde and ChenzhouRenmin4th Hospitals.\u003c/p\u003e\n\u003cp\u003eD. AUROC performance in brain tumor versus cerebrovascular disease discrimination. Left panel: external test on XiangyaChangde dataset. Right panel: external validation on ChenzhouRenmin4th dataset. OmiMRI_I1 through OmiMRI_I4 represent the same model utilizing one to four MRIs for inference.\u003c/p\u003e\n\u003cp\u003eE. AUROC performance in glioma versus meningioma discrimination. Left panel: external validation on XiangyaChangde dataset. Right panel: external test on ChenzhouRenmin4th dataset.\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-8651933/v1/ae68c98f90d7fda9d255d5b4.png"},{"id":100761195,"identity":"cbd9a162-a153-485e-854e-9c71f125fe18","added_by":"auto","created_at":"2026-01-21 07:37:16","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":2227820,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003ePracticability of OmiMRI in data-limited discrimination of glioblastoma (GBM) versus brain metastasis (MET).\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAbbreviations: area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPRC).\u003c/p\u003e\n\u003cp\u003eA. Experimental setup for data-limited classification. A small-scale brain MRI dataset was selected from patients with GBM or MET from HunanXiangya. Seven neuroradiologists specializing in brain tumors independently diagnosed 727 randomly selected samples from the test set.\u003c/p\u003e\n\u003cp\u003eB. AUROC and AUPRC performance across groups for GBM vs. MET classification. The horizontal axis indicates the same OmiMRI model using one to four MRIs in inference. Data are presented as mean ± SEM.\u003c/p\u003e\n\u003cp\u003eC. ROC curves for the three groups in GBM vs. MET classification. Solid lines represent mean values; shaded areas indicate 95% confidence intervals (CI). Panels from left to right: combined input with one to four MRIs.\u003c/p\u003e\n\u003cp\u003eD. PR curves for the three groups in GBM vs. MET classification.\u003c/p\u003e\n\u003cp\u003eE. Saliency visualizations of OmiMRI for a combined input with three MRIs from a GBM patient. Left panel: original brain MRIs. Right panel: corresponding saliency heatmaps.\u003c/p\u003e\n\u003cp\u003eF. Saliency visualizations of OmiMRI for a combined input with three MRIs from a MET patient.\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-8651933/v1/5af8bee762169152355cd297.png"},{"id":100762074,"identity":"616e1c22-8257-4e8c-9039-4bca19000c2e","added_by":"auto","created_at":"2026-01-21 07:52:59","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":15708523,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8651933/v1/f239c5ec-a2c1-4ef7-bb1b-e820dcd5b355.pdf"},{"id":100761260,"identity":"62b25bf7-f54a-4286-b5a7-a70891b67ac5","added_by":"auto","created_at":"2026-01-21 07:38:32","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":1415273,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryMaterials.docx","url":"https://assets-eu.researchsquare.com/files/rs-8651933/v1/113edae62ae07c01c570b074.docx"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"OmiMRI: A Clinical-adaptive AI Framework for Format-Free Interpretation of Heterogeneous Brain MRIs","fulltext":[{"header":"Introduction","content":"\u003cp\u003eMagnetic resonance imaging (MRI) is a powerful, non-invasive technique for visualizing both the structural and functional details of the brain\u003csup\u003e1\u0026ndash;3\u003c/sup\u003e. It remains the gold standard for diagnosing numerous neurological disorders\u003csup\u003e4\u0026ndash;6\u003c/sup\u003e, including glioblastoma (GBM), brain metastases (MET), cerebrovascular diseases, neurodegenerative disorders, neuropsychiatric disorders, and pathological aging. Among these, GBM and MET are the most prevalent malignant brain tumors worldwide, with a median survival time of less than 16 months\u003csup\u003e7\u0026ndash;9\u003c/sup\u003e, posing significant challenges for personalized diagnosis. Artificial intelligence (AI) holds the potential to fully leverage the value of MRIs. However, most existing AI models are still in the developmental stage and face an \u0026quot;AI chasm\u0026quot; before they can be effectively deployed in clinical practice\u003csup\u003e10\u0026ndash;12\u003c/sup\u003e. Despite remarkable progress, most AI models implicitly assume that heterogeneity in clinical MRI should be minimized or corrected through standardization or harmonization\u003csup\u003e13,14\u003c/sup\u003e. This assumption fundamentally conflicts with real-world clinical practice, where variability is intrinsic, informative, and often unavoidable.\u003c/p\u003e\n\u003cp\u003eTo fully realize the potential of AI models, efficient data and robust frameworks are essential. MRI plays a crucial role in modern healthcare, with approximately 100 to 150 million scans performed annually worldwide by the 2020s\u003csup\u003e15\u0026ndash;17\u003c/sup\u003e. This has led to the accumulation of vast quantities of clinical brain MRIs, including numerous open-source datasets available through platforms such as The Cancer Imaging Archive (TCIA)\u003csup\u003e18\u003c/sup\u003e, NeuroImaging Tools \u0026amp; Resources Collaboratory (NITRC)\u003csup\u003e19\u003c/sup\u003e, UK Biobank\u003csup\u003e20\u003c/sup\u003e, and OpenNeuro\u003csup\u003e21\u003c/sup\u003e. These resources hold immense value for data-driven AI models aimed at enhancing the neuroimage analysis and neurological disorders diagnosis\u003csup\u003e22\u0026ndash;24\u003c/sup\u003e. Clinical MRIs are extremely heterogeneous and inherently personalized, exhibiting significant variations in scan sequences, slice resolution, spacing, and slice count. However, most advanced brain MRI models remain confined to oversimplified settings with meticulously preprocessed data that adhere to strict formatting requirements (Fig. S1). The notable rigidity of these models prevents them from effectively handling the substantial heterogeneity observed in clinical settings\u003csup\u003e25,26\u003c/sup\u003e. Despite efforts to address issues such as site heterogeneity and modality missing\u003csup\u003e13,14,27,28\u003c/sup\u003e, a substantial gap remains between research development and clinical deployment\u003csup\u003e10\u0026ndash;12\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eInstead of enforcing a canonical input format, we propose that clinically meaningful AI systems should adapt to whatever data are available for each individual patient. To bridge this gap,\u0026nbsp;here we present OmiMRI, a unified, attribute-agnostic framework capable of processing arbitrary combinations of MRI inputs (Fig. 1, Fig. S2). Unlike conventional \u0026quot;format-rigid\u0026quot; approaches that require strict input standardization which often necessitating data exclusion or extensive preprocessing, OmiMRI adopts a \u0026quot;format-free\u0026quot; paradigm designed for direct training and inference on highly variable clinical data. OmiMRI supports various advanced backbones for initial feature extraction, followed by self-attention mechanisms or dynamic weighting to fuse variable-length feature representations. When provided with two or more combined MRIs, OmiMRI consistently outperforms conventional single-input models across 15 brain MRI tasks, with performance improves steadily as more sequences are incorporated.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWe assembled an unprecedentedly large and diverse multi-center brain MRI cohort, integrating nearly all publicly available brain tumor and cerebrovascular disease resources alongside a proprietary dataset spanning 16 hospitals. The in-house data cover over 88% of total patient cases for four of the six major brain diseases, substantially enriching resources for high-quality brain MRI research. Through continual learning experiments, we rapidly established a model prototype using single-center data, then progressively expanded the dataset to include more centers and samples. This enabled iterative version evolution, with performance improving across external validation datasets. In a challenging data-limited clinical task, OmiMRI matches or surpasses senior neuroradiologists in discriminating GBM from brain MET, demonstrating robust performance and clinically interpretable attention patterns. Together, these results establish OmiMRI as a practical, scalable paradigm for clinically adaptive brain MRI analysis under real-world heterogeneity.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003e\u003cstrong\u003eOmiMRI: a practical AI framework for heterogeneous clinical environments\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOmiMRI is specifically designed to address the challenges posed by the extreme heterogeneity of MRIs encountered in clinical practice. This AI framework is \u0026quot;format-free\u0026quot; and effectively handles arbitrary MRI inputs, accommodating significant variations in MRI count, sequence combinations, and imaging properties (Fig. 1A, Fig. 2A). The complexity of clinical MRIs arises from three primary sources. First, there is a broad array of MRI sequences routinely used, with over 12 common types, including T1-weighted (T1w), T2-weighted (T2w), contrast-enhanced T1 (T1c), T2 fluid-attenuated inversion recovery (T2f), magnetic resonance angiography (MRA), and apparent diffusion coefficient (ADC). Second, acquisition parameters vary considerably across scans, with differences in scan axis, slice resolution, slice spacing, and slice count (Fig. 2A). Finally, clinical practice introduces inherent combinatorial complexity, as MRI examinations are tailored to individual diagnostic needs, resulting in significant inter-patient variability (Fig. 2A).\u003c/p\u003e\n\u003cp\u003eConventional AI strategies typically employ \u0026quot;format-rigid\u0026quot; modeling, which requires standardized inputs and consequently forcing the exclusion of non-conforming data (Fig. 1B). They also necessitate training distinct models for every unique combination of input modalities (Fig. 1C). In contrast, OmiMRI introduces a \u0026quot;format-free\u0026quot; modeling paradigm designed to directly process arbitrary, heterogeneous MRI combinations (Fig. 1B). OmiMRI uses a single architecture capable of handling varying input subsets, ranging from single T1w scans to complex multi-modal combinations. This design significantly reduces preprocessing labor and enhances practical data utility compared to conventional methods (Fig. 1C).\u003c/p\u003e\n\u003cp\u003eOmiMRI functions as a universal interface that supports a series of pretrained backbones (such as BrainSegFounder, BrainIAC, or standard convolutional neural networks (CNNs)) for initial feature extraction. To address the dimensionality mismatch caused by heterogeneous inputs, OmiMRI projects the extracted features into a normalized space prior to fusion (Fig. 1C). The system then applies task-specific aggregation strategies\u0026mdash;using Transformer-based self-attention for linear head tasks (classification and regression) or dynamic weighting for dense prediction tasks (segmentation)\u0026mdash;enabling a single, scalable platform to handle diverse brain MRI tasks across institutions.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eQuantifying the heterogeneity of clinical brain MRIs\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eClinical brain MRI data exhibit significant heterogeneity compared to research-centered datasets, primarily due to variations in scanning protocols driven by diagnostic and treatment requirements. In research settings, MRI acquisition protocols are designed with specific research objectives, often simplifying the scanning and analysis process by tightly controlling MRI parameters. Consequently, research-driven MRI data are typically structured and standardized, showing high consistency across subjects (Fig. 2A). However, MRI datasets from different research centers or with varying objectives often display substantial heterogeneity, reflecting differences in scanning and preprocessing protocols.\u003c/p\u003e\n\u003cp\u003eIn contrast, the primary goal of clinical brain MRI is to diagnose and treat diseases. The selection of scanning sequences, counts, and parameters is guided by clinical needs. Both six open-source datasets and in-house brain MRI data from 16 hospitals show extreme heterogeneity (Fig. 2A). This heterogeneity is evident in two aspects. First, there is significant individual variation; each patient\u0026apos;s disease condition is unique, making clinical brain MRI data inherently personalized. Second, there is substantial variability across institutions and disease types. Different brain diseases often require distinct combinations of MRI sequences to accurately reveal disease status. Additionally, scanning protocols vary widely among institutions.\u003c/p\u003e\n\u003cp\u003eEven widely used MRI sequences, such as T1w and T2w, exhibit considerable complexity in clinical practice. For example, in the RocklandSample research dataset, T1w data demonstrate perfectly consistent slice parameters\u0026mdash;including view, resolution, number, and step\u0026mdash;across all subjects (Fig. 2B). However, in clinical practice, such as in the HunanXiangya dataset, T1w and T2w slice parameters can vary across as many as 120 different combinations (Fig. 2B, C). There are also substantial differences across centers and diseases; for instance, the T1w scan slice parameters in the ABIDE-1 dataset show 12 combinations, while the T2w scan slice parameters in the YunnanRenmin1st dataset exhibit 24 different combinations.\u003c/p\u003e\n\u003cp\u003eMost importantly, the vast majority of brain MRIs are generated in clinical settings, providing more abundant, valuable, and complicated resources than curated research datasets. The immense potential of clinical brain MRI resources remains untapped, highlighting the need for AI frameworks capable of addressing their extreme heterogeneity.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePerformance of OmiMRI and comparison with classic models\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo evaluate OmiMRI\u0026apos;s capability to handle arbitrary combinations of input sequences and achieve effective feature fusion, we progressively increased the number of MRI sequences used during testing. Initially, only T1w were used, followed by the addition of T2w, T1c, and T2f sequences. In volume and image classification tasks (LGG vs. HGG), the AUROC increased significantly as more MRI sequences were incorporated (Fig. 3A, B). Similarly, in segmentation tasks, the mean Dice score for each label showed significant improvement with the inclusion of additional sequences (Fig. 3C, D; Fig. S3).\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;To evaluate OmiMRI\u0026rsquo;s versatility, we tested the model across a broad spectrum of MRI tasks. These tasks were grouped into three main categories: volume classification and regression (six tasks, Fig. 3E), image segmentation (six tasks, Fig. 3F), and image classification (three tasks, Fig. 3G). In traditional AI frameworks, each model is typically tailored to a specific task type. For example, the Conv3D TPN model is used for volume classification and regression, while ResNet50 is used for image classification. In contrast, the OmiMRI framework demonstrated its versatility by successfully handling 15 tasks across these three categories, highlighting its broad applicability. For each task, we trained models using various combinations of MRI inputs. During testing, we categorized the results based on the number of MRIs: OmiMRI_I1, OmiMRI_I2, and OmiMRI_I3+ represent the same model using combined inputs with 1, 2, and three or more MRIs for inference, respectively. The results showed that performance consistently improved as the number of input MRIs increased across all downstream tasks. Compared to the nine classical models that accept fixed input with a single MRI, OmiMRI_I2 and OmiMRI_I3+ exhibited a comprehensive performance advantage (Fig. 3E-G).\u003c/p\u003e\n\u003cp\u003eWe further demonstrate that OmiMRI outperforms fixed-input models in both performance and practicality. Using identical backbone architectures, datasets, and training conditions, OmiMRI consistently surpasses four medical imaging foundation models as well as two natural video‑pretrained models when processing 2 to 4 input MRIs. Compared to fixed-input models, OmiMRI achieves superior F1 scores in 94.4% of comparisons, with an average absolute improvement of 0.038 \u0026plusmn; 0.031 (Table 1, Table S1).\u003c/p\u003e\n\u003cp\u003eThese results confirm OmiMRI\u0026apos;s \u0026quot;attribute-agnostic\u0026quot; nature and its ability to integrate diverse MRI inputs, thereby enhancing diagnostic performance and making it better suited for clinical practice.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEvolving OmiMRI with large-scale heterogeneous MRIs\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo assess the continual learning capacity of OmiMRI, we incrementally incorporated datasets from different centers into training, thereby generating progressively evolving model versions. OmiMRI processes arbitrary MRI combinations with minimal preprocessing requirements, facilitating direct utilization of multi-center clinical data and seamless model evolution. We curated a large-scale, multicenter brain MRI cohort covering six brain tumor and cerebrovascular diseases (Fig. 4A). Publicly available data were collected from 18 datasets on TCIA, BraTS2023, OpenNeuro, and ATLAS, totaling 30,938 3D MRIs. In-house data were obtained from 16 hospitals, with 60,843 3D MRIs in total. Public datasets showed substantial imbalance, with gliomas accounting for 74% of cases and a paucity of multiple disease types from the same center (Fig. 4A; Fig. S4A). In contrast, our in-house dataset is greatly balanced across six diseases, ranging from 11% for meningiomas to 23% for intracranial aneurysms (Fig. 4A; Fig. S4B). The superior diversity and balance of the in-house MRI dataset, together with its comprehensive disease coverage within single centers, provided a strong foundation for OmiMRI development.\u003c/p\u003e\n\u003cp\u003eWhile multicenter datasets greatly expand scale and diversity, they also introduce pronounced heterogeneity. OmiMRI demonstrated strong capacity to accommodate such heterogeneity, thereby fully leveraging the advantages of large-scale MRI resources. In practice, this enables an iterative clinical development paradigm: initial prototypes (V1.0) can be rapidly built using data from a limited number of centers, followed by scaling, systematic optimization, and physician feedback to yield successive versions (V1.x) that continually evolve in real-world settings (Fig. 4B). For example, starting with HunanXiangya data to establish OmiMRI V1.0, we successively incorporated additional centers to produce OmiMRI V1.1, V1.2, and V1.3 (Fig. 4C). In two validation datasets (XiangyaChangde and ChenzhouRenmin4th), across two tasks (brain tumors vs. cerebrovascular diseases and glioma vs. meningioma), AUROC improved significantly with evolving version (Fig. 4D, E).\u003c/p\u003e\n\u003cp\u003eTogether, these results demonstrate OmiMRI\u0026rsquo;s ability to evolve through continual integration of heterogeneous MRI data, progressively enhancing its performance and clinical applicability.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePracticality of OmiMRI in data-limited scenarios\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo evaluate the practicality of OmiMRI in data-limited scenarios, we conducted glioblastoma (GBM) and metastasis (MET) discrimination tasks, comparing its performance with that of neuroradiologists specializing in brain tumors. We used a small brain MRI dataset from HunanXiangya, comprising 546 subjects (GBM: 254; MET: 292) and 3,221 MRIs (GBM: 1,409; MET: 1,812) (Fig. 5A). During testing, the area under the receiver operating characteristic curve (AUROC) increased with the number of input MRIs across all three groups (Junior (n = 4), Senior (n = 3), and OmiMRI (n = 3)). With three MRIs, OmiMRI and the Senior group demonstrated comparable performance in AUROC (OmiMRI: 0.895, 95% CI: 0.878\u0026ndash;0.906; Senior: 0.876, 95% CI: 0.843\u0026ndash;0.904; \u003cem\u003eP\u003c/em\u003e \u0026gt; 0.05) and the area under the precision-recall curve (AUPRC) (OmiMRI: 0.933, 95% CI: 0.922\u0026ndash;0.939; Senior: 0.912, 95% CI: 0.880\u0026ndash;0.936; \u003cem\u003eP\u003c/em\u003e \u0026gt; 0.05). With four MRIs, OmiMRI achieved comparable AUROC (OmiMRI: 0.931, 95% CI: 0.909\u0026ndash;0.948; Senior: 0.907, 95% CI: 0.873\u0026ndash;0.927; \u003cem\u003eP\u003c/em\u003e \u0026gt; 0.05) and significantly higher AUPRC than the Senior group (OmiMRI: 0.973, 95% CI: 0.964\u0026ndash;0.979; (Senior: AUPRC: 0.930, 95% CI: 0.904\u0026ndash;0.945; \u003cem\u003eP\u003c/em\u003e \u0026lt; 0.05) (Fig. 5B\u0026ndash;D; Fig. S5\u0026ndash;S8). OmiMRI also clearly outperformed the Junior group when three or four combined MRIs were used in the inputs (Fig. 5B\u0026ndash;D; Fig. S5\u0026ndash;S8).\u003c/p\u003e\n\u003cp\u003eIn distinguishing GBM from MET, OmiMRI\u0026rsquo;s attention heatmaps closely corresponded with key radiological and pathological features. For GBM, OmiMRI exhibited broad, high-attention values throughout the entire tumor region, including the tumor core and surrounding edema, consistent with its infiltrative growth pattern and peripheral hypercellular margins (Fig. 5E). In contrast, for MET, the attention heatmap was more compact and confined to the tumor core, with well-defined boundaries that matched the pathological presentation of metastatic edema margins (Fig. 5F). Furthermore, across varying numbers, sequences, and imaging planes of input MRI combinations, OmiMRI consistently localized lesion areas within each MRI scan (Fig. 5E, F; Supplementary Fig. S9).\u003c/p\u003e\n\u003cp\u003eOmiMRI achieves expert-level performance in discriminating GBM from MET using minimal training data. It identifies critical tumor features that align with established clinical knowledge across diverse MRI sequences and scan views. This concordance enhances interpretability and underscores its clinical relevance. These results demonstrate the strong performance and interpretability of OmiMRI, highlighting its practicality for clinical applications.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eExisting efforts to address clinical MRI variability can be broadly categorized into three paradigms: (i) harmonization-based approaches that attempt to normalize data distributions across sites\u003csup\u003e13,14\u003c/sup\u003e; (ii) missing-modality approaches that assume a predefined modality set and explicitly impute absent inputs\u003csup\u003e27,28\u003c/sup\u003e; and (iii) adaptive frameworks that treat heterogeneity itself as a modeling objective. OmiMRI belongs to the third category. Importantly, only this paradigm directly aligns with real-world clinical MRI acquisition, where input availability is inherently patient-specific and often unpredictable.\u003c/p\u003e\n\u003cp\u003eOmiMRI presents a practical and unified AI framework for real-world clinical neuroimaging. It accepts arbitrary MRI inputs, enabling great practicality in complex hospital settings. By remaining agnostic to sequence type, count, and spatial properties, OmiMRI eliminates the notion of a canonical input space altogether. The system evolves with large-scale multi-center data, and matches or surpasses expert-level diagnosis in data-limited scenarios.\u003c/p\u003e\n\u003cp\u003eRather than treating variability in MRIs as a problem to be corrected\u003csup\u003e29,30\u003c/sup\u003e, we designed OmiMRI to inherently embrace this heterogeneity as a fundamental characteristic of clinical data. The framework operates on the principle that any combination of available MRI sequences (regardless of type, count, or spatial properties) should be seamlessly integrable for meaningful feature extraction and inference. By adopting this attribute-agnostic philosophy, OmiMRI eliminates the need for extensive data harmonization while preserving the rich, complementary information embedded across different MRI sequences. This approach effectively bridges the gap between the rigid requirements of conventional AI models and the dynamic, personalized nature of real-world clinical data.\u003c/p\u003e\n\u003cp\u003eIt is essential to distinguish OmiMRI from recent segmentation foundation models, such as \u003cstrong\u003eMedSAM\u003c/strong\u003e\u003csup\u003e31\u003c/sup\u003e and \u003cstrong\u003eSAM-Med3D\u003c/strong\u003e\u003csup\u003e32\u003c/sup\u003e. While these generalist models have achieved state-of-the-art (SOTA) performance on standardized benchmarks, they are primarily \u0026quot;format-rigid\u0026quot;\u0026mdash;often restricted to one imaging input with specific channels (e.g., 3-channel RGB) or requiring manual prompting for optimal performance. In clinical practice, where multi-sequence availability is inherently inconsistent (e.g., missing T1c or T2f), these models may require sequence-specific retraining or data imputation. OmiMRI addresses this by serving as a \u0026quot;format-free\u0026quot; fusion framework. As demonstrated by our integration of various backbones (including SOTA brain MRI foundation models of \u003cstrong\u003eBrainSegFounder\u003c/strong\u003e and \u003cstrong\u003eBrainIAC\u003c/strong\u003e), OmiMRI does not compete with these architectures but rather empowers them, unlocking their diagnostic potential in heterogeneous clinical environments without necessitating rigid input standardization.\u003c/p\u003e\n\u003cp\u003eA key advantage of OmiMRI lies in its ability to accept arbitrary MRI inputs. Conventional AI models often require stringent data curation, leading to significant exclusion of clinically valuable scans\u003csup\u003e33,34\u003c/sup\u003e. In contrast, OmiMRI\u0026apos;s attribute-agnostic design accommodates the inherent heterogeneity of clinical MRIs. This eliminates the need for extensive preprocessing and manual data harmonization, thereby reducing labor costs and minimizing biases introduced by subjective data selection. The framework makes full use of available clinical data, preserving informative variations often lost in standardized research datasets.\u003c/p\u003e\n\u003cp\u003eThe performance gains of OmiMRI stem from its capacity to integrate information from multiple MRI inputs\u003csup\u003e35,36\u003c/sup\u003e. Through self-attention\u0026ndash;based fusion, the framework learns to identify and align complementary features across multiple modalities and imaging views, enabling more comprehensive characterization of pathological signatures than single-sequence models. Importantly, the attention patterns not only enhance performance but also provide clinically interpretable visualizations that correlate with established radiological knowledge.\u003c/p\u003e\n\u003cp\u003eOmiMRI also possesses an intrinsic capability to handle missing modalities, which is a common challenge in clinical practice\u003csup\u003e27,28\u003c/sup\u003e. Unlike approaches that rely on explicit imputation techniques or require specialized architectures for different input combinations, OmiMRI dynamically adapts its feature extraction and fusion processes based on the available inputs. This flexibility offers a distinct practical advantage over fixed-input models, facilitating reliable inference even when certain sequences are unavailable.\u003c/p\u003e\n\u003cp\u003eBeyond individual tasks, OmiMRI supports an incremental development paradigm that aligns with real-world research and clinical implementation timelines. Initial prototypes can be rapidly established using data from a limited number of centers. As more institutions contribute data, the framework systematically incorporates new heterogeneous information in the \u0026quot;format-free\u0026quot; manner, demonstrating progressive performance improvement across external validation sets. This process yields progressive performance improvements on external validation datasets without requiring complete retraining, making the framework particularly suitable for large-scale, multi-center collaboration.\u003c/p\u003e\n\u003cp\u003eWhile OmiMRI demonstrates substantial clinical applicability, several promising directions for future development remain. First, self-supervised pre-training of 2D and 3D backbones on large-scale neuroimaging data could further improve feature learning and diagnostic accuracy. Second, while OmiMRI currently focuses on structural MRI (sMRI), integrating functional MRI (fMRI) would provide complementary dynamic brain information. The integration of both sMRI and fMRI data within the OmiMRI framework could significantly enhance the diagnosis of neuropsychiatric disorders. Third, the framework could be extended to incorporate additional clinical data types, such as textual diagnostic reports and pathological whole-slide images. Such multimodal integration would better reflect real-world clinical workflows and strengthen diagnostic performance across diverse medical scenarios. We view these directions as natural extensions of the framework rather than prerequisites for its current clinical utility.\u003c/p\u003e"},{"header":"Materials and methods","content":"\u003cp\u003e\u003cstrong\u003eCuration of brain MRI dataset\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe Ethics Committee of Xiangya Hospital of Central South University (HunanXiangya) approved this retrospective analysis of anonymous patient data (No. 2025101736). HunanXiangya led and coordinated the entire data curation process. Data collection and analysis in this retrospective study were conducted anonymously.\u003c/p\u003e\n\u003cp\u003eWe retrospectively collected brain MRI data from 16 Chinese hospitals, covering six major brain diseases. The curated large-scale dataset spans brain tumors (BT) and cerebrovascular diseases (CVD), where BT includes glioma, meningioma, and metastasis, while CVD comprises infarction, aneurysm, and hemorrhage. The data collection period extended from 2005 to 2025. Following quality control and de-identification procedures, all data were pseudonymized with recoded identifiers before being included in the analysis. We selected 18 high-quality datasets focusing on the six brain diseases from over 100 publicly available sources\u003csup\u003e37\u003c/sup\u003e, including ten TCIA datasets, five BraTS2023 datasets, two OpenNeuro datasets, and ATLAS R2.0 stroke dataset.\u003c/p\u003e\n\u003cp\u003eAll brain MRIs underwent skull stripping to protect patient privacy and minimize background interference. Subsequently, the intensity values of each MRI were normalized to the 0-255 range based on its 5th and 99.9th percentile values.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAnalysis of brain MRI attributes\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eMulti-parametric MRI sequences provide complementary information, revealing distinct structural and functional characteristics of the brain. Key sequences include: T1-weighted (T1w) imaging for anatomical detail; T2-weighted (T2w) for fluid and pathology detection; contrast-enhanced T1 (T1c) for assessing blood-brain barrier disruption; T2-fluidattenuated inversion recovery (FLAIR) (T2f) for suppressing cerebrospinal fluid signal; apparent diffusion coefficient (ADC) maps for quantifying water diffusion; diffusion-weighted imaging (DWI) for acute ischemia evaluation; susceptibility-weighted imaging (SWI) for detecting blood and calcifications; magnetic resonance angiography (MRA) for vascular visualization; and proton density (PD) weighting for tissue characterization.\u003c/p\u003e\n\u003cp\u003eSpatially, MRIs are categorized as isotropic or anisotropic. Isotropic volumes are defined by a single spatial parameter of voxel resolution (\u003cem\u003evoxel_reso\u003c/em\u003e, mm/voxel), with values in this study predominantly falling within 0.75 \u0026lt; \u003cem\u003evoxel_reso\u003c/em\u003e \u0026le; 1.0. For anisotropic acquisitions, four slice parameters were analyzed: slice resolution (\u003cem\u003eslice_reso\u003c/em\u003e, mm/pixel), categorized as \u0026le; 0.5, 0.5\u0026ndash;1.0, or 1.0\u0026ndash;2.0; slice step (\u003cem\u003eslice_step\u003c/em\u003e, mm), categorized as \u0026le; 0.5, 0.5\u0026ndash;1.0, 1.0\u0026ndash;2.0, 2.0\u0026ndash;4.0, or \u0026gt; 4.0; slice number (\u003cem\u003eslice_num\u003c/em\u003e), categorized as \u0026le; 20, 20\u0026ndash;40, 40\u0026ndash;80, or \u0026gt; 80; and slice view, categorized as axial, sagittal, or coronal.\u003c/p\u003e\n\u003cp\u003eIn clinical practice, the selection of MRI sequences and acquisition parameters is tailored to individual diagnostic needs, resulting in substantial variation in both the number and type of MRIs acquired per subject. Inter-subject consistency (i.e., the inverse of heterogeneity) was assessed for each dataset and attribute by first calculating the standard deviation (std) of MRI counts per subject. Consistency score was defined as \u003cem\u003eexp(\u0026ndash;std)\u003c/em\u003e, ranging from 0 to 1, with lower values indicating greater heterogeneity. For instance, in a research dataset where every subject undergoes identical isotropic T1w scanning, the std for both the T1w and voxel_reso attributes is zero, yielding a consistency score of 1. In contrast, clinical datasets exhibit lower consistency scores, reflecting their intrinsic heterogeneity arising from scanning individualization.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDeveloping OmiMRI models\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eRationale and motivation.\u0026nbsp;\u003c/strong\u003eSeveral studies have attempted to address the complexity of clinical MRI data, such as aligning T1w MRI scans from different centers into a harmonized space\u003csup\u003e13,14\u003c/sup\u003e, and generating missing MRI sequences from acquired sequences\u0026mdash;for instance, generating T2w MRI from T1w MRI\u003csup\u003e27,28\u003c/sup\u003e. However, site heterogeneity and modality missing represent only the tip of the iceberg when it comes to the complexity of heterogeneous MRI data in clinical settings. The development of attribute-agnostic AI models capable of handling extremely heterogeneous MRI data remains a critically important yet largely unexplored frontier. In the field of multimodal large language models (MM-LLM), a dominant and cutting-edge approach involves using multiple encoders to extract features from different modalities separately, then usually integrating these features using a Transformer-based model, enabling cross-modal interaction and fusion\u003csup\u003e38,39\u003c/sup\u003e. MM-LLM hold the capability to process inputs of varying lengths and quantities dynamically and adaptively. This functional adaptability characteristic bears resemblance to the \u0026ldquo;attribute-agnostic\u0026rdquo; nature inherent in clinical brain MRI practice, providing the foundational inspiration for the current study. However, due to the distinct characteristics of biomedical images, these existing MM-LLM models cannot be directly applied to MRI-based diagnosis.\u003c/p\u003e\n\u003cp\u003eWe conceptualize combined multi-sequence MRI inputs as a form of multimodal data representation. Within the OmiMRI framework, each individual MRI sequence undergoes independent feature extraction via general-purpose backbone encoders. The resulting features are then projected into a normalized embedding space to eliminate dimensional and semantic disparities. A transformer-based fusion module subsequently enables dynamic feature integration through self-attention mechanisms, effectively capturing cross-sequence dependencies. OmiMRI framework achieves intrinsic adaptability to clinical MRI heterogeneity without requiring predefined input specifications or modality alignment.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eArchitecture.\u0026nbsp;\u003c/strong\u003eWe began by systematically analyzing prevailing architectural paradigms for brain MRI analysis (Fig. S1). Current methodologies span a spectrum of input designs: from single 2D slice inputs where models process individual anatomical slices (Fig. S1A)\u003csup\u003e31,37\u003c/sup\u003e, to 3D volume analysis where entire MRI volumes are processed holistically (Fig. S1B)\u003csup\u003e32,37\u003c/sup\u003e. For multi-sequence (T1w, T2w, T1c, and T2f) integration, two predominant strategies have emerged: 4D concatenation approaches where co-registered isotropic sequences are stacked along a channel dimension (Fig. S1C)\u003csup\u003e40,41\u003c/sup\u003e; and modality-specific encoder designs where dedicated pathways process predefined sequences followed by feature fusion (Fig. S1D)\u003csup\u003e42\u0026ndash;44\u003c/sup\u003e. These multi-sequence methods share fundamental constraints\u0026mdash;they require isotropic, co-registered inputs and predefined modality combinations. These requirements fundamentally limit their applicability to real-world clinical data, where heterogeneity in acquisition protocols and frequent modality missing are inherent characteristics.\u003c/p\u003e\n\u003cp\u003eIn response, we developed OmiMRI as a unified, attribute-agnostic framework that accepts arbitrary MRI combinations (Fig. 1, Fig. S2). The architecture employs general backbone networks to independently extract features for all MRIs in the combined input, which are linearly projected into a unified embedding space to normalize dimensional and semantic disparities. A transformer-based fusion module then enables cross-sequence feature interaction through multi-head self-attention mechanisms, effectively capturing long-range dependencies across variable-input contexts. This design eliminates requirements for isotropic resolution, spatial co-registration, or predefined modality sets, thereby directly processing native clinical MRI data without structural harmonization. The framework\u0026apos;s modular organization further ensures extensibility to emerging network architectures and additional neuroimaging modalities.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eInputs.\u0026nbsp;\u003c/strong\u003eA fundamental capability of OmiMRI is processing combined inputs with variable MRI counts, where the set size \u003cem\u003eS\u003c/em\u003e represents the number of MRIs per input. To construct these combined inputs, for a subject with \u003cem\u003eN\u003c/em\u003e available MRI scans, the number of possible distinct subsets of size \u003cem\u003eS\u003c/em\u003e is given by the binomial coefficient \u003cem\u003eC(N, S)\u003c/em\u003e. We evaluated \u003cem\u003eS\u003c/em\u003e values of 1, 2, 3, and 4. To ensure dataset balance across different set sizes, when \u003cem\u003eC(N, S)\u003c/em\u003e exceeded 2\u0026times;\u003cem\u003eN\u003c/em\u003e, we randomly sampled a maximum of 2\u0026times;\u003cem\u003eN\u003c/em\u003e combinations. Within each combined input, the \u003cem\u003eS\u003c/em\u003e MRIs were presented in a randomized order at every iteration to prevent the model from learning any spurious correlations based on the sequence order. This approach of combined input construction directly addresses the heterogeneity in the number of MRIs per patient encountered in clinical practice.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePreprocessing.\u0026nbsp;\u003c/strong\u003eTo enable batch processing in the volume MRI experiments, input data were formatted into a 6-dimensional tensor with layout \u003cem\u003eBSDCHW\u003c/em\u003e. Here, \u003cem\u003eB\u003c/em\u003e denotes batch size; \u003cem\u003eS\u003c/em\u003e represents set size; \u003cem\u003eD\u003c/em\u003e indicates depth, corresponding to the count of effective slices per MRI (with effective slices defined as containing \u0026gt;10% non-zero pixels). The depth dimension was standardized to \u003cem\u003eD\u003c/em\u003e through either resampling or zero-padding. In preliminary experiments, we evaluated \u003cem\u003eD\u003c/em\u003e values of 24, 32, 48, and 64, ultimately selecting \u003cem\u003eD\u003c/em\u003e=32 for all subsequent experiments as it provided the optimal balance between performance and computational efficiency.\u003cem\u003e\u0026nbsp;C\u003c/em\u003e represents channels (fixed at \u003cem\u003eC\u003c/em\u003e=3 following conversion of grayscale MRI slices to RGB format to leverage pre-trained weights from natural image/video models). \u003cem\u003eH\u003c/em\u003e and \u003cem\u003eW\u003c/em\u003e denote slice height and width, standardized through resizing. Height \u003cem\u003eH\u003c/em\u003e and width \u003cem\u003eW\u003c/em\u003e were standardized through resizing. Preliminary tests with shapes of 128\u0026times;128, 192\u0026times;192, and 256\u0026times;256 led to the optimal selection of 192\u0026times;192 for classification and regression and 256\u0026times;256 for segmentation. This spatial shape preprocessing pipeline standardizes the \u003cem\u003eDHW\u003c/em\u003e dimensions while remaining agnostic to MRI sequence type or scanning orientation. For 2D MRI slice experiments, depth dimension \u003cem\u003eD\u003c/em\u003e=1\u003cem\u003e.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eGroup batching.\u0026nbsp;\u003c/strong\u003eIn practical model development, batch processing is essential for achieving efficient and effective training. We implemented a group batching strategy stratified by the set size \u003cem\u003eS\u003c/em\u003e. Fist, organize the datasets into distinct groups according to their \u003cem\u003eS\u003c/em\u003e values, with each group containing samples of identical set sizes. Then, given a predefined batch size \u003cem\u003eB\u003c/em\u003e, we randomly select \u003cem\u003eB\u003c/em\u003e samples from the same group to form a 6-dimensional input following the \u003cem\u003eBSDCHW\u003c/em\u003e layout. During training, each iteration randomly loads batches from different set-size groups. This group batching approach effectively handles data with varying set sizes while maintaining computational efficiency during deep model training and inference.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTraining.\u0026nbsp;\u003c/strong\u003eOmiMRI integrates dynamically swappable backbones and task-specific heads to achieve substantial versatility. The dimensions of the raw features (\u003cem\u003eDx\u003c/em\u003e and \u003cem\u003eLx\u003c/em\u003e) depend on the backbone architecture and input data shape. Following feature normalization, the feature dimension is standardized to \u003cem\u003eD0\u003c/em\u003e = 128, with a channel count of \u003cem\u003eLy\u003c/em\u003e = 10. During feature concatenation, the \u003cem\u003eS\u003c/em\u003e feature maps are stacked along the channel dimension. After self-attention computation, a 512-dimensional feature vector is obtained, which is subsequently passed through a linear layer for final classification or regression.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIn preliminary 3D experiments, we evaluated various backbones with pretrained parameters from the MMAction2 platform, including \u003cem\u003eResNet50_SlowOnly_TPN\u003c/em\u003e, \u003cem\u003eResNet50_SlowFast\u003c/em\u003e\u003csup\u003e45\u003c/sup\u003e\u003cem\u003e, ResNet34_R2plus1D\u003c/em\u003e\u003csup\u003e46\u003c/sup\u003e\u003cem\u003e, ViT_Video_small\u003c/em\u003e\u003csup\u003e47\u003c/sup\u003e\u003cem\u003e,\u0026nbsp;\u003c/em\u003eand \u003cem\u003eSwinT_Video_small\u003c/em\u003e\u003csup\u003e48\u003c/sup\u003e. \u003cem\u003eResNet50_SlowOnly_TPN\u003c/em\u003e was selected as the best backbone for OmiMRI, offering the optimal balance between performance and efficiency. For 2D experiments, we tested backbones with pretrained parameters from Torchvision, including \u003cem\u003eResNet18, ResNet50, MobileNet v2\u003c/em\u003e, and \u003cem\u003eViT\u003c/em\u003e. \u003cem\u003eResNet50\u003c/em\u003e was ultimately chosen as the best backbone for OmiMRI based on its balanced performance. In segmentation tasks, we evaluated \u003cem\u003eDeepLab v3\u003c/em\u003e networks\u003csup\u003e49,50\u003c/sup\u003e with \u003cem\u003eResNet18, ResNet50, HRNet v2\u003c/em\u003e, and \u003cem\u003eMobileNet v2\u003c/em\u003e backbones during preliminary experiments. The \u003cem\u003eDeepLab v3\u003c/em\u003e architecture with a \u003cem\u003eResNet50\u003c/em\u003e backbone was selected for OmiMRI\u0026apos;s segmentation tasks.\u003c/p\u003e\n\u003cp\u003eThese selections were guided by systematic evaluation of each architecture\u0026apos;s compatibility with our heterogeneous MRI data characteristics. \u003cem\u003eResNet50\u003c/em\u003e variants consistently demonstrated superior adaptability to the varied spatial resolutions and sequence parameters in our clinical datasets, while maintaining computational feasibility for hospital deployment. Our current implementation represents an initial configuration within OmiMRI\u0026apos;s flexible architecture. The framework\u0026apos;s modular design intentionally accommodates future integration of emerging backbone architectures as they become available. We anticipate that with substantially expanded dataset scales, transformer-based architectures such as \u003cem\u003eViT\u003c/em\u003e and \u003cem\u003eSwinT\u003c/em\u003e variants may demonstrate advantages in capturing complex patterns across heterogeneous data. This forward compatibility ensures continuous performance improvement while maintaining consistency in clinical deployment, allowing the framework to evolve with advancing model architectures and growing data resources.\u003c/p\u003e\n\u003cp\u003eAll experiments were conducted on a Biomedical Computing Platform utilizing NVIDIA A800 GPUs with 80GB memory. We employed consistent general parameter settings across all tasks: \u003cem\u003eAdamW\u003c/em\u003e optimizer with a base learning rate of 0.0001, \u003cem\u003eL2\u003c/em\u003e weight decay of 0.0005, and early stopping patience of 6 epochs. For loss functions, we used \u003cem\u003eCrossEntropyLoss\u003c/em\u003e for classification tasks, SmoothL1Loss for regression tasks, and \u003cem\u003eFocalLoss\u003c/em\u003e for segmentation tasks to address class imbalance.\u003c/p\u003e\n\u003cp\u003eTask-specific parameters were optimized for each experimental paradigm. For 3D volumetric analysis, we used a batch size of 16 with 2 gradient accumulation steps, training for 1,000-10,000 steps with validation every 200 steps. For 2D classification tasks, we set a batch size of 32 with 2 gradient accumulation steps, training for 200-5,000 steps with validation every 20 steps. For 2D segmentation tasks, we maintained a batch size of 32 but increased gradient accumulation to 4 steps, training for 2,000-20,000 steps with validation every 500 steps to accommodate the increased computational demands of pixel-wise prediction.\u003c/p\u003e\n\u003cp\u003eThis stratified parameter configuration achieves an optimal balance between model performance and computational efficiency across different task types and data modalities.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTasks and experiments\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3D Volume Analysis.\u0026nbsp;\u003c/strong\u003eFor volume classification, multiple public datasets were utilized. The glioma versus metastasis (GLI vs. MET) and glioma versus meningioma (GLI vs. MEN) tasks were constructed using the BraTS2023 dataset. Low-grade glioma versus high-grade glioma (LGG vs. HGG) classification was performed with the BraTS2020 dataset. The LGG versus glioblastoma (GBM) task incorporated both TCGA-LGG and TCGA-GBM datasets, while isocitrate dehydrogenase (IDH) mutation prediction was developed using TCGA-LGG and UPENN-GBM datasets. For the volume regression task, brain age prediction was implemented with the IXI dataset. For all tasks, the dataset was split into training, validation, and test sets at the subject level using a 7:1:2 ratio.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTo establish performance baselines, we compared OmiMRI against several established 3D architectures based on MMAction2 platform, including \u003cem\u003eResNet50_SlowOnly_TPN\u003c/em\u003e (Conv3D TPN), \u003cem\u003eResNet34_R2plus1D\u003c/em\u003e (Conv2D+1D), and \u003cem\u003eResNet50_SlowFast\u003c/em\u003e (Conv3D SlowFast) models. These conventional approaches typically process single MRI sequences and require standardized input formats, serving as representative benchmarks for current volume-based analysis methods.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2D Image Analysis.\u0026nbsp;\u003c/strong\u003eFor 2D image classification, slice-level datasets were constructed from BraTS2023 and BraTS2020 volumetric MRI data. Effective slices were extracted along the axial plane, with tumor-positive slices defined as containing tumor regions exceeding 20 pixels, and tumor-negative slices containing no tumor pixels. To maintain dataset balance, all positive slices from each subject were retained, while negative slices were randomly subsampled at 25%. The glioma recognition (GLI Recog) task distinguished glioma-positive from glioma-negative slices; meningioma recognition (MEN Recog) differentiated meningioma-positive from glioma-negative slices; and LGG vs. HGG classification was performed using tumor-positive slices from BraTS2020. For 2D segmentation tasks, we constructed slice-level datasets from the BraTS2023 glioma and meningioma collections. Segmentation datasets were annotated with three distinct tumor subregions: non-enhancing tumor core (NETC), surrounding non-enhancing FLAIR hyperintensity (SNFH), and enhancing tumor (ET). For all tasks, the dataset was split into training, validation, and test sets at the subject level using a 7:1:2 ratio.\u003c/p\u003e\n\u003cp\u003eFor comparative evaluation in 2D tasks, we implemented several widely-used architectures. In classification tasks, \u003cem\u003eResNet50\u003c/em\u003e, \u003cem\u003eMobileNet v2\u003c/em\u003e, and \u003cem\u003eResNet18\u003c/em\u003e based on Torchvision served as baseline models, representing current standard approaches for single-image analysis. For segmentation tasks, we benchmarked against DeepLab v3 with \u003cem\u003eResNet50\u003c/em\u003e, \u003cem\u003eHRNet v2\u003c/em\u003e, and \u003cem\u003eMobileNet v2\u003c/em\u003e backbones, which constitute representative contemporary methods for medical image segmentation.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eOmiMRI vs. fixed-input models:\u003c/strong\u003e Most of existing models requires fixed -input. Six different backbone architectures were tested, among which CT-FM\u003csup\u003e51\u003c/sup\u003e, BrainSegFounder\u003csup\u003e52\u003c/sup\u003e, BrainIAC\u003csup\u003e53\u003c/sup\u003e, and BrainLaMIM\u003csup\u003e41\u003c/sup\u003e are pretrained medical imaging foundation models, while Conv2D+1D and Conv3D-TPN are models pretrained on natural videos.\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003eEach Fixed model can only process a fixed number of inputs; thus, four separate fixed models were trained for scenarios with 1, 2, 3, and 4 input MRIs respectively. In contrast, the OmiMRI framework requires training only a single model to perform inference with any arbitrary number of inputs. We evaluated the models on two 3D classification tasks: glioma grading (LGG versus HGG) and IDH mutation prediction.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eOmiMRI evolving experiment.\u0026nbsp;\u003c/strong\u003eIn the evolving experiments, large-scale multi-center data were grouped by institution and sequentially introduced into the training set to produce progressively evolved model versions. The substantial dataset from HunanXiangya served as the foundation, yielding the initial model version V1.0. Subsequent versions were developed by incrementally expanding the training data: V1.1 incorporated data from HubeiTCM, YichangZhongxin, YueyangLinxiangRenmin, and ChenzhouRenmin3rd; V1.2 added data from HunanUnivMedicine, HunanRenmin, HuaihuaTCM, and ZhuzhouChalingRenmin; and V1.3 further integrated data from YueyangZhongxin, YunnanRenmin1st, Changsha4th, and Zhuzhou331.\u003c/p\u003e\n\u003cp\u003eAll data from these centers were divided into training, validation, and test sets at the subject level using a 7:1:2 ratio. External validation was performed using data from XiangyaChangde and ChenzhouRenmin4th, where 10% of the data supported model fine-tuning, 70% was used for metric evaluation, and 20% was held in reserve. The evolving experiments evaluated OmiMRI on two clinical discrimination tasks: brain tumor versus cerebrovascular disease, and glioma versus meningioma.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData-limited GBM vs. MET.\u0026nbsp;\u003c/strong\u003eGlioblastoma (GBM) and brain metastasis (MET) represent two highly malignant brain tumors with overlapping radiological presentations but distinct pathogenesis and treatment regimens. Accurate discrimination between them carries significant implications for surgical planning and therapeutic strategy.\u003c/p\u003e\n\u003cp\u003eWe curated a clinically challenging, small-scale brain MRI dataset from HunanXiangya Hospital, comprising patients diagnosed between 2010 and 2025. Following rigorous manual quality control, the final cohort included 546 subjects and 3,221 MRIs. For OmiMRI development, we constructed combination samples containing one to four MRIs per input. These samples were divided into training, validation, and test sets using a subject-level split ratio of 7:1:2. Saliency heatmaps for the test dataset were generated using the Grad-CAM method, providing visual explanations of the model\u0026apos;s focus areas during decision-making.\u003c/p\u003e\n\u003cp\u003eIn the human expert evaluation, seven neuroradiologists (four junior and three seniors, with clinical experience ranging from 3 to 22 years) specializing in brain tumors independently diagnosed 727 randomly selected samples from the test set. The expert evaluation dataset consisted of 200 samples with a single MRI, 200 samples with 2\u0026ndash;3 MRIs per combined input, and 127 samples with four MRIs per combined input. Each expert assessed the probability of GBM and MET for every sample using a discrete scoring system with increments of 0.1, with the constraint that the sum of the probability scores for GBM and MET must equal 1.0.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEvaluation.\u0026nbsp;\u003c/strong\u003eThe area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC) were employed for classification tasks. Mean absolute error (MAE) was used for the brain age prediction regression task. The Dice score was adopted for segmentation tasks. For each task, the OmiMRI model was independently trained three times, and the average evaluation metrics were calculated.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eData \u0026amp; Code\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eAvailability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003ePublic datasets used in this study are available from their original sources. Aggregated statistics, cohort summaries, and selected derived data supporting the findings are provided in the Supplementary Information; additional data related to this study are being further curated for consolidated reference and may be made available for academic research upon reasonable request and subject to institutional approval.\u003c/p\u003e\n\u003cp\u003eThe implementation of the OmiMRI framework, including model architectures, training scripts, and evaluation pipelines, will be made publicly available upon publication via a permanent online repository (e.g., GitHub or Zenodo). Pretrained model weights and configuration files corresponding to the experiments reported in this study will also be released to facilitate reproducibility and future benchmarking.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe thank the Biomedical Computing Platform of National Biomedical Imaging Center at Peking University for providing computational resources. We express our gratitude to the neuroradiologists who participated in the glioblastoma (GBM) and metastasis (MET) discrimination tasks. This work is supported by Beijing Natural Science Foundation (7254542, JQ24023, F251020), the Beijing Municipal Science and Technology Commission Project (Grant No. Z231100006623010), China Postdoctoral Science Foundation (8206301107), and the Peking University Medicine Plus X Pilot Program \u0026ndash; Artificial Intelligence and Medical Development Initiative (BMU2025YXXLHAIYX020).\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eEisenstein, M. Pushing the limits of MRI brain imaging. \u003cem\u003eNat Methods\u003c/em\u003e \u003cstrong\u003e21\u003c/strong\u003e, 1975\u0026ndash;1979 (2024).\u003c/li\u003e\n\u003cli\u003eHess, C. P. MRI of the Brain: What Is Driving Innovation in 2023? \u003cem\u003eRadiology\u003c/em\u003e \u003cstrong\u003e308\u003c/strong\u003e, e231657 (2023).\u003c/li\u003e\n\u003cli\u003eLohrke, J. \u003cem\u003eet al.\u003c/em\u003e 25 Years of Contrast-Enhanced MRI: Developments, Current Challenges and Future Perspectives. \u003cem\u003eAdv Ther\u003c/em\u003e \u003cstrong\u003e33\u003c/strong\u003e, 1\u0026ndash;28 (2016).\u003c/li\u003e\n\u003cli\u003eAli, S. S. A., Memon, K., Yahya, N. \u0026amp; Khan, S. Deep learning frameworks for MRI-based diagnosis of neurological disorders: a systematic review and meta-analysis. \u003cem\u003eArtif Intell Rev\u003c/em\u003e \u003cstrong\u003e58\u003c/strong\u003e, 171 (2025).\u003c/li\u003e\n\u003cli\u003eParizel, P. M. \u003cem\u003eet al.\u003c/em\u003e Magnetic Resonance Imaging of the Brain. in \u003cem\u003eClinical MR Imaging\u003c/em\u003e (eds Reimer, P., Parizel, P. M., Meaney, J. F. M. \u0026amp; Stichnoth, F. A.) 107\u0026ndash;195 (Springer Berlin Heidelberg, Berlin, Heidelberg, 2010). doi:10.1007/978-3-540-74504-4_2.\u003c/li\u003e\n\u003cli\u003eYang, Z. \u003cem\u003eet al.\u003c/em\u003e Brain aging patterns in a large and diverse cohort of 49,482 individuals. \u003cem\u003eNat Med\u003c/em\u003e \u003cstrong\u003e30\u003c/strong\u003e, 3015\u0026ndash;3026 (2024).\u003c/li\u003e\n\u003cli\u003eSchaff, L. R. \u0026amp; Mellinghoff, I. K. Glioblastoma and Other Primary Brain Malignancies in Adults: A Review. \u003cem\u003eJAMA\u003c/em\u003e \u003cstrong\u003e329\u003c/strong\u003e, 574\u0026ndash;587 (2023).\u003c/li\u003e\n\u003cli\u003eTanzhu, G. \u003cem\u003eet al.\u003c/em\u003e Metastatic brain tumors: from development to cutting‐edge treatment. \u003cem\u003eMedComm\u003c/em\u003e \u003cstrong\u003e6\u003c/strong\u003e, e70020 (2025).\u003c/li\u003e\n\u003cli\u003eZhang, Y. \u003cem\u003eet al.\u003c/em\u003e Glioblastoma-associated macrophages in glioblastoma: from their function and mechanism to therapeutic advances. \u003cem\u003eCancer Gene Ther\u003c/em\u003e \u003cstrong\u003e32\u003c/strong\u003e, 595\u0026ndash;607 (2025).\u003c/li\u003e\n\u003cli\u003eReyna, M. A., Nsoesie, E. O. \u0026amp; Clifford, G. D. Rethinking Algorithm Performance Metrics for Artificial Intelligence in Diagnostic Medicine. \u003cem\u003eJAMA\u003c/em\u003e \u003cstrong\u003e328\u003c/strong\u003e, 329\u0026ndash;330 (2022).\u003c/li\u003e\n\u003cli\u003eRosenthal, J. T., Beecy, A. \u0026amp; Sabuncu, M. R. Rethinking clinical trials for medical AI with dynamic deployments of adaptive systems. \u003cem\u003enpj Digit. Med.\u003c/em\u003e \u003cstrong\u003e8\u003c/strong\u003e, 252 (2025).\u003c/li\u003e\n\u003cli\u003eZhang, A., Xing, L., Zou, J. \u0026amp; Wu, J. C. Shifting machine learning for healthcare from development to deployment and from models to data. \u003cem\u003eNat. Biomed. Eng\u003c/em\u003e \u003cstrong\u003e6\u003c/strong\u003e, 1330\u0026ndash;1345 (2022).\u003c/li\u003e\n\u003cli\u003eCackowski, S., Barbier, E. L., Dojat, M. \u0026amp; Christen, T. ImUnity: A generalizable VAE-GAN solution for multicenter MR image harmonization. \u003cem\u003eMed Image Anal\u003c/em\u003e \u003cstrong\u003e88\u003c/strong\u003e, 102799 (2023).\u003c/li\u003e\n\u003cli\u003eRoca, V. \u003cem\u003eet al.\u003c/em\u003e IGUANe: A 3D generalizable CycleGAN for multicenter harmonization of brain MR images. \u003cem\u003eMed Image Anal\u003c/em\u003e \u003cstrong\u003e99\u003c/strong\u003e, 103388 (2025).\u003c/li\u003e\n\u003cli\u003eMartella, M., Lenzi, J. \u0026amp; Gianino, M. M. Diagnostic Technology: Trends of Use and Availability in a 10-Year Period (2011\u0026ndash;2020) among Sixteen OECD Countries. \u003cem\u003eHealthcare (Basel)\u003c/em\u003e \u003cstrong\u003e11\u003c/strong\u003e, 2078 (2023).\u003c/li\u003e\n\u003cli\u003eSmith-Bindman, R. \u003cem\u003eet al.\u003c/em\u003e Trends in Use of Medical Imaging in US Health Care Systems and in Ontario, Canada, 2000-2016. \u003cem\u003eJAMA\u003c/em\u003e \u003cstrong\u003e322\u003c/strong\u003e, 843\u0026ndash;856 (2019).\u003c/li\u003e\n\u003cli\u003eP\u0026auml;r, K. Number of MRI Scans Per Year Worldwide: Overview of global MRI utilization. \u003cem\u003ehttps://collectiveminds.health/\u003c/em\u003e (2024).\u003c/li\u003e\n\u003cli\u003eClark, K. \u003cem\u003eet al.\u003c/em\u003e The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. \u003cem\u003eJ Digit Imaging\u003c/em\u003e \u003cstrong\u003e26\u003c/strong\u003e, 1045\u0026ndash;1057 (2013).\u003c/li\u003e\n\u003cli\u003eKennedy, D. N., Haselgrove, C., Riehl, J., Preuss, N. \u0026amp; Buccigrossi, R. The NITRC image repository. \u003cem\u003eNeuroImage\u003c/em\u003e \u003cstrong\u003e124\u003c/strong\u003e, 1069\u0026ndash;1073 (2016).\u003c/li\u003e\n\u003cli\u003eSudlow, C. \u003cem\u003eet al.\u003c/em\u003e UK Biobank: An Open Access Resource for Identifying the Causes of a Wide Range of Complex Diseases of Middle and Old Age. \u003cem\u003ePLoS Med\u003c/em\u003e \u003cstrong\u003e12\u003c/strong\u003e, e1001779 (2015).\u003c/li\u003e\n\u003cli\u003eMarkiewicz, C. J. \u003cem\u003eet al.\u003c/em\u003e The OpenNeuro resource for sharing of neuroscience data. \u003cem\u003eeLife\u003c/em\u003e \u003cstrong\u003e10\u003c/strong\u003e, e71774 (2021).\u003c/li\u003e\n\u003cli\u003eSun, Y., Wang, L., Li, G., Lin, W. \u0026amp; Wang, L. A foundation model for enhancing magnetic resonance images and downstream segmentation, registration and diagnostic tasks. \u003cem\u003eNat. Biomed. Eng\u003c/em\u003e 1\u0026ndash;18 (2024) doi:10.1038/s41551-024-01283-7.\u003c/li\u003e\n\u003cli\u003eWang, L. \u003cem\u003eet al.\u003c/em\u003e A lifespan-generalizable skull-stripping model for magnetic resonance images that leverages prior knowledge from brain atlases. \u003cem\u003eNat. Biomed. Eng\u003c/em\u003e 1\u0026ndash;16 (2025) doi:10.1038/s41551-024-01337-w.\u003c/li\u003e\n\u003cli\u003eElyounssi, S. \u003cem\u003eet al.\u003c/em\u003e Addressing artifactual bias in large, automated MRI analyses of brain development. \u003cem\u003eNat Neurosci\u003c/em\u003e \u003cstrong\u003e28\u003c/strong\u003e, 1787\u0026ndash;1796 (2025).\u003c/li\u003e\n\u003cli\u003eMoor, M. \u003cem\u003eet al.\u003c/em\u003e Foundation models for generalist medical artificial intelligence. \u003cem\u003eNature\u003c/em\u003e \u003cstrong\u003e616\u003c/strong\u003e, 259\u0026ndash;265 (2023).\u003c/li\u003e\n\u003cli\u003eHe, J. \u003cem\u003eet al.\u003c/em\u003e The practical implementation of artificial intelligence technologies in medicine. \u003cem\u003eNature Medicine\u003c/em\u003e \u003cstrong\u003e25\u003c/strong\u003e, 30\u0026ndash;36 (2019).\u003c/li\u003e\n\u003cli\u003eHu, X. B. \u003cem\u003eet al.\u003c/em\u003e AutoGAN-Synthesizer: Neural Architecture Search for Cross-Modality MRI Synthesis. \u003cem\u003eMedical Image Computing and Computer Assisted Intervention (MICCAI 2022)\u003c/em\u003e \u003cstrong\u003e13436\u003c/strong\u003e, 397\u0026ndash;409 (2022).\u003c/li\u003e\n\u003cli\u003eLi, H. W. \u003cem\u003eet al.\u003c/em\u003e DiamondGAN: Unified Multi-modal Generative Adversarial Networks for MRI Sequences Synthesis. \u003cem\u003eMedical Image Computing and Computer Assisted Intervention (MICCAI 2019)\u003c/em\u003e \u003cstrong\u003e11767\u003c/strong\u003e, 795\u0026ndash;803 (2019).\u003c/li\u003e\n\u003cli\u003eOrlhac, F. \u003cem\u003eet al.\u003c/em\u003e How can we combat multicenter variability in MR radiomics? Validation of a correction procedure. \u003cem\u003eEur Radiol\u003c/em\u003e \u003cstrong\u003e31\u003c/strong\u003e, 2272\u0026ndash;2280 (2021).\u003c/li\u003e\n\u003cli\u003eDa-Ano, R., Visvikis, D. \u0026amp; Hatt, M. Harmonization strategies for multicenter radiomics investigations. \u003cem\u003ePhys. Med. Biol.\u003c/em\u003e \u003cstrong\u003e65\u003c/strong\u003e, 24TR02 (2020).\u003c/li\u003e\n\u003cli\u003eMa, J. \u003cem\u003eet al.\u003c/em\u003e Segment anything in medical images. \u003cem\u003eNat Commun\u003c/em\u003e \u003cstrong\u003e15\u003c/strong\u003e, 654 (2024).\u003c/li\u003e\n\u003cli\u003eWang, H. \u003cem\u003eet al.\u003c/em\u003e SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images. Preprint at https://doi.org/10.48550/arXiv.2310.15161 (2024).\u003c/li\u003e\n\u003cli\u003eHagler, D. J. \u003cem\u003eet al.\u003c/em\u003e Image processing and analysis methods for the Adolescent Brain Cognitive Development Study. \u003cem\u003eNeuroImage\u003c/em\u003e \u003cstrong\u003e202\u003c/strong\u003e, 116091 (2019).\u003c/li\u003e\n\u003cli\u003eAlfaro-Almagro, F. \u003cem\u003eet al.\u003c/em\u003e Image processing and Quality Control for the first 10,000 brain imaging datasets from UK Biobank. \u003cem\u003eNeuroImage\u003c/em\u003e \u003cstrong\u003e166\u003c/strong\u003e, 400\u0026ndash;424 (2018).\u003c/li\u003e\n\u003cli\u003eZhao, H., Cai, H. \u0026amp; Liu, M. Transformer based multi-modal MRI fusion for prediction of post-menstrual age and neonatal brain development analysis. \u003cem\u003eMedical Image Analysis\u003c/em\u003e \u003cstrong\u003e94\u003c/strong\u003e, 103140 (2024).\u003c/li\u003e\n\u003cli\u003eLiu, Z., Wei, J., Li, R. \u0026amp; Zhou, J. Learning multi-modal brain tumor segmentation from privileged semi-paired MRI images with curriculum disentanglement learning. \u003cem\u003eComputers in Biology and Medicine\u003c/em\u003e \u003cstrong\u003e159\u003c/strong\u003e, 106927 (2023).\u003c/li\u003e\n\u003cli\u003eSu, F. \u003cem\u003eet al.\u003c/em\u003e From Slices to Volumes: A Scalable Pipeline for Developing General-Purpose Brain MRI Foundation Models. Preprint at https://doi.org/10.1101/2025.04.12.25325728 (2025).\u003c/li\u003e\n\u003cli\u003eYin, S. \u003cem\u003eet al.\u003c/em\u003e A Survey on Multimodal Large Language Models. \u003cem\u003eNational Science Review\u003c/em\u003e \u003cstrong\u003e11\u003c/strong\u003e, nwae403 (2024).\u003c/li\u003e\n\u003cli\u003eZhang, D. \u003cem\u003eet al.\u003c/em\u003e MM-LLMs: Recent Advances in MultiModal Large Language Models. in \u003cem\u003eFindings of the Association for Computational Linguistics ACL 2024\u003c/em\u003e 12401\u0026ndash;12430 (Association for Computational Linguistics, Bangkok, Thailand and virtual meeting, 2024). doi:10.18653/v1/2024.findings-acl.738.\u003c/li\u003e\n\u003cli\u003eLiu, H. \u003cem\u003eet al.\u003c/em\u003e M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities. Preprint at http://arxiv.org/abs/2303.05302 (2023).\u003c/li\u003e\n\u003cli\u003eChen, M. \u003cem\u003eet al.\u003c/em\u003e Medical image foundation models in assisting diagnosis of brain tumors: a pilot study. \u003cem\u003eEur Radiol\u003c/em\u003e https://doi.org/10.1007/s00330-024-10728-1 (2024) doi:10.1007/s00330-024-10728-1.\u003c/li\u003e\n\u003cli\u003eShi, J. \u003cem\u003eet al.\u003c/em\u003e PASSION: Towards Effective Incomplete Multi-Modal Medical Image Segmentation with Imbalanced Missing Rates. Preprint at https://doi.org/10.48550/arXiv.2407.14796 (2024).\u003c/li\u003e\n\u003cli\u003ePipoli, V. \u003cem\u003eet al.\u003c/em\u003e IM-Fuse: A Mamba-Based Fusion Block for Brain Tumor Segmentation with Incomplete Modalities. in \u003cem\u003eMedical Image Computing and Computer Assisted Intervention \u0026ndash; MICCAI 2025\u003c/em\u003e (eds Gee, J. C. et al.) 225\u0026ndash;235 (Springer Nature Switzerland, Cham, 2026).\u003c/li\u003e\n\u003cli\u003eZeng, Z., Peng, Z., Yang, X. \u0026amp; Shen, W. Missing as Masking: Arbitrary Cross-Modal Feature Reconstruction for Incomplete Multimodal Brain Tumor Segmentation. in \u003cem\u003eMedical Image Computing and Computer Assisted Intervention \u0026ndash; MICCAI 2024\u003c/em\u003e (eds Linguraru, M. G. et al.) 424\u0026ndash;433 (Springer Nature Switzerland, Cham, 2024).\u003c/li\u003e\n\u003cli\u003eFeichtenhofer, C., Fan, H., Malik, J. \u0026amp; He, K. SlowFast Networks for Video Recognition. in \u003cem\u003e2019 IEEE/CVF International Conference on Computer Vision (ICCV)\u003c/em\u003e 6201\u0026ndash;6210 (2019). doi:10.1109/ICCV.2019.00630.\u003c/li\u003e\n\u003cli\u003eTran, D. \u003cem\u003eet al.\u003c/em\u003e A Closer Look at Spatiotemporal Convolutions for Action Recognition. \u003cem\u003e2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition\u003c/em\u003e 6450\u0026ndash;6459 (2017).\u003c/li\u003e\n\u003cli\u003eTong, Z., Song, Y., Wang, J. \u0026amp; Wang, L. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. in \u003cem\u003eAdvances in Neural Information Processing Systems\u003c/em\u003e (eds Koyejo, S. et al.) vol. 35 10078\u0026ndash;10093 (Curran Associates, Inc., 2022).\u003c/li\u003e\n\u003cli\u003eLiu, Z. \u003cem\u003eet al.\u003c/em\u003e Video Swin Transformer. in \u003cem\u003e2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)\u003c/em\u003e 3192\u0026ndash;3201 (2022). doi:10.1109/CVPR52688.2022.00320.\u003c/li\u003e\n\u003cli\u003eChen, L.-C., Papandreou, G., Schroff, F. \u0026amp; Adam, H. Rethinking Atrous Convolution for Semantic Image Segmentation. Preprint at https://doi.org/10.48550/arXiv.1706.05587 (2017).\u003c/li\u003e\n\u003cli\u003eChen, L.-C., Zhu, Y., Papandreou, G., Schroff, F. \u0026amp; Adam, H. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. Preprint at https://doi.org/10.48550/arXiv.1802.02611 (2018).\u003c/li\u003e\n\u003cli\u003ePai, S. \u003cem\u003eet al.\u003c/em\u003e Vision Foundation Models for Computed Tomography. Preprint at https://doi.org/10.48550/arXiv.2501.09001 (2025).\u003c/li\u003e\n\u003cli\u003eCox, J. \u003cem\u003eet al.\u003c/em\u003e BrainSegFounder: Towards 3D foundation models for neuroimage segmentation. \u003cem\u003eMedical Image Analysis\u003c/em\u003e \u003cstrong\u003e97\u003c/strong\u003e, 103301 (2024).\u003c/li\u003e\n\u003cli\u003eTak, D. \u003cem\u003eet al.\u003c/em\u003e A foundation model for generalized brain MRI analysis. 2024.12.02.24317992 Preprint at https://doi.org/10.1101/2024.12.02.24317992 (2024). \u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Table 1","content":"\u003cp\u003e\u003cstrong\u003eTable 1. Comparison of F1-scores between OmiMRI models and Fixed models\u003c/strong\u003e\u003c/p\u003e\n\u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"650\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"3\" style=\"width: 123px;\"\u003e\n \u003cp\u003eModel name (backbone)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"3\" style=\"width: 55px;\"\u003e\n \u003cp\u003eModel type\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"4\" style=\"width: 236px;\"\u003e\n \u003cp\u003eGlioma grading (LGG vs. HGG)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"4\" style=\"width: 236px;\"\u003e\n \u003cp\u003eIDH mutation prediction\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"4\" style=\"width: 236px;\"\u003e\n \u003cp\u003eCount of input MRIs\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"4\" style=\"width: 236px;\"\u003e\n \u003cp\u003eCount of input MRIs\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 59px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 59px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 59px;\"\u003e\n \u003cp\u003e3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 59px;\"\u003e\n \u003cp\u003e4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 59px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 59px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 59px;\"\u003e\n \u003cp\u003e3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 59px;\"\u003e\n \u003cp\u003e4\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" style=\"width: 123px;\"\u003e\n \u003cp\u003eCT-FM (ResNet)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 55px;\"\u003e\n \u003cp\u003eFixed\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.768\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.778\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.783\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.772\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.559\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.717\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.804\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.834\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 55px;\"\u003e\n \u003cp\u003eOmi\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.779\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.804\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.814\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.830\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.556\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.727\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.805\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.841\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" style=\"width: 123px;\"\u003e\n \u003cp\u003eBrainSegFounder (SwinT)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 55px;\"\u003e\n \u003cp\u003eFixed\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.740\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.748\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.746\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.751\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.569\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.741\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.809\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.829\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 55px;\"\u003e\n \u003cp\u003eOmi\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.777\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.825\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.839\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.846\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.556\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.731\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.818\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.848\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" style=\"width: 123px;\"\u003e\n \u003cp\u003eBrainIAC (ViT)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 55px;\"\u003e\n \u003cp\u003eFixed\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.766\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.748\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.736\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.788\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.572\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.676\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.731\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.780\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 55px;\"\u003e\n \u003cp\u003eOmi\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.733\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.783\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.791\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.791\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.598\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.746\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.809\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.826\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" style=\"width: 123px;\"\u003e\n \u003cp\u003eBrainLaMIM (ViT)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 55px;\"\u003e\n \u003cp\u003eFixed\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.747\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.795\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.769\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.779\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.503\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.652\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.714\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.711\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 55px;\"\u003e\n \u003cp\u003eOmi\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.776\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.817\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.824\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.827\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.551\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.692\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.775\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.800\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" style=\"width: 123px;\"\u003e\n \u003cp\u003eConv2D+1D (ResNet34)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 55px;\"\u003e\n \u003cp\u003eFixed\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.810\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.828\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.818\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.811\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.693\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.769\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.832\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.872\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 55px;\"\u003e\n \u003cp\u003eOmi\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.817\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.833\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.834\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.836\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.640\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.768\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.847\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.875\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" style=\"width: 123px;\"\u003e\n \u003cp\u003eConv3D-TPN (ResNet50)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 55px;\"\u003e\n \u003cp\u003eFixed\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.814\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.783\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.845\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.869\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.674\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.775\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.827\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.845\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 55px;\"\u003e\n \u003cp\u003eOmi\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.843\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.885\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.913\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.919\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e0.639\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.784\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.851\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 59px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.874\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eSix different backbone architectures were tested. Bold text indicates instances where the OmiMRI model achieved superior performance compared to the Fixed model with the same backbone and input. For each task, both the OmiMRI models and Fixed models utilized identical training, validation, and test datasets. For each model, the parameter counts of the encoder, Omi-Neck \u0026amp; Head, and Fixed-Neck \u0026amp; Head are presented in Table S1.\u0026nbsp;Note that for the \u0026apos;Fixed\u0026apos; models, four separate models had to be trained and deployed for each input combination, whereas OmiMRI used a single model for all scenarios.\u003c/p\u003e\n"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-8651933/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8651933/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eClinical brain MRI analysis faces a fundamental challenge: bridging the gap between oversimplified research developments and the inherent heterogeneity of real-world clinical practice. Quantifying this gap, our analysis of 26 MRI attributes across 22 clinical datasets reveals substantial heterogeneity across institutions and patients. Current AI tools typically require rigid input formats, necessitating extensive data exclusion or preprocessing that severely limits their real-world utility.\u003c/p\u003e\n\u003cp\u003eHere we present OmiMRI, a unified, format-free framework designed to bridge this gap by enabling adaptive processing of arbitrary MRI combinations. Rather than strictly defining a standalone architecture, OmiMRI functions as a universal framework that integrates diverse pretrained 2D/3D convolutional and Transformer-based networks as feature encoders. Through a self-attention mechanism and dynamic weighting to fuse features from variable inputs, OmiMRI decouples clinical performance from rigid input specifications and enables adaptive processing of arbitrary MRI combinations.\u003c/p\u003e\n\u003cp\u003eAcross 15 diverse classification, segmentation, and regression tasks, OmiMRI demonstrates robust input-scaling capabilities, yielding significant improvements over traditional fixed-input models. Notably, OmiMRI outperforms advanced medical imaging foundation models (e.g., BrainIAC and BrainSegFounder) in 94.4% of comparisons involving 2–4 input MRIs under consistent experimental conditions. Furthermore, the framework exhibits continuous performance gains through the incremental incorporation of multi-center data. In a clinically demanding, data-limited task distinguishing glioblastoma from metastasis, OmiMRI achieved diagnostic performance matching or exceeding that of senior neuroradiologists (AUROC 0.931 vs. 0.907, \u003cem\u003eP\u003c/em\u003e \u0026gt; 0.05; AUPRC 0.973 vs. 0.930, \u003cem\u003eP\u003c/em\u003e \u0026lt; 0.05), while providing interpretable attention maps aligned with radiological landmarks. Together, these results establish OmiMRI as a clinically adaptive AI paradigm that transform format-rigid modeling into flexible, expert-level systems capable of embracing the heterogeneity of real-world patient data.\u003c/p\u003e","manuscriptTitle":"OmiMRI: A Clinical-adaptive AI Framework for Format-Free Interpretation of Heterogeneous Brain MRIs","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-01-21 07:16:26","doi":"10.21203/rs.3.rs-8651933/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-biomedical-engineering","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"natbiomedeng","sideBox":"Learn more about [Nature Biomedical Engineering](http://www.nature.com/natbiomedeng/)","snPcode":"41551","submissionUrl":"https://mts-natbiomedeng.nature.com/cgi-bin/main.plex","title":"Nature Biomedical Engineering","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Research","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"44e06aa4-8604-4cf5-b87d-4005f5e4a40b","owner":[],"postedDate":"January 21st, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":61456637,"name":"Biological sciences/Computational biology and bioinformatics/Image processing"},{"id":61456638,"name":"Biological sciences/Biological techniques/Imaging/Magnetic resonance imaging"},{"id":61456639,"name":"Physical sciences/Mathematics and computing/Computational science"}],"tags":[],"updatedAt":"2026-01-21T07:16:26+00:00","versionOfRecord":[],"versionCreatedAt":"2026-01-21 07:16:26","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8651933","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8651933","identity":"rs-8651933","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}