Evaluation of a Multimodal Convolutional Neural Network-Based Approach for DICOM Files Classification

Evaluation of a Multimodal Convolutional Neural Network-Based Approach for DICOM Files Classification | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Evaluation of a Multimodal Convolutional Neural Network-Based Approach for DICOM Files Classification Vicent Mabirizi, Simon Kawuma, William Wasswa This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7727424/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The Digital Imaging and Communication in Medicine (DICOM) standard preserve both pixel-level image data and clinically relevant metadata. However, conventional deep learning pipelines for medical image classification often discard this metadata by converting DICOM files into formats such as PNG or JPEG, leading to information loss and potential bias. This study evaluated the performance of DICOMFusionNet, a multimodal convolutional neural network (CNN) developed to natively process DICOM files by integrating both image data and embedded metadata, in comparison with widely used transfer learning models. A dataset of 1000 pediatric chest radiographs (425 tuberculosis-positive and 575 controls) from Epicentre, Mbarara Regional Referral Hospital, was used. Images were pre-processed to enhance pulmonary visibility, and relevant metadata fields were normalized and one-hot encoded for integration. DICOMFusionNet was benchmarked against Inception V3, VGG16, VGG19 and ResNet50, all requiring DICOM-to-PNG conversion. Performance was evaluated using accuracy, precision, recall, and F1-score. An ablation study assessed the contribution of metadata to classification performance. DICOMFusionNet achieved superior performance with a test accuracy of 92.3% and F1-score of 0.91, outperforming Inception V3 (86.7%), VGG16 (85.4%), VGG19 (85.9%), and ResNet50 (87.1%). The ablation study revealed a significant drop in accuracy (87.8%) and F1-score (0.85) when metadata was excluded, highlighting its critical role in predictive performance. DICOMFusionNet demonstrates that preserving both image and metadata in medical imaging tasks yields more accurate and context-aware classification. This multimodal approach reduces bias, enhance generalization, and provides a promising framework for clinical decision support in diagnostic imaging. DICOMFusionNet Multimodal Deep Learning Medical Image Classification Metadata Integration 1. Introduction This paper presents an evaluation of a deep learning model for Digital Imaging and Communication in Medicine (DICOM) image classification, previously developed in our earlier work(Vicent et al., 2025 ), hereafter referred to as DICOMFusionNet. The model was developed to natively process DICOM files using a multimodal Convolutional Neural Network (CNN) architecture that integrates both pixel-level image data and embedded metadata. This approach enhances classification accuracy and reduces bias by eliminating the need for format conversion from DICOM to other formats such as PNG and JPEG, a process that can alter image content or strip ancillary information, thereby affecting downstream model performance (Mayer et al., 2025 ; Varma, 2012 ). The original study achieved improved results on two datasets including chest X-ray for tuberculosis detection and magnetic resonance imaging (MRI) scans for brain tumor classification. However, although DICOMFusionNet outperformed conventional approaches in internal evaluations, questions remain about its comparative performance and efficiency relative to CCN-based transferring models, including Inception V3 (Meena et al., 2023 ), the Vision Geometry Group (VGG) networks (Veni & Manjula, 2022 ), and ResNet50 (Koonce, 2021 ), typically require a conversion pipeline from DICOM to formats like PNG or JPEG (Kim & Kim, 2022 ; Mabirizi et al., 2025 ). To answer this question, we conducted a comparative experiment to evaluate the performance of the DICOMFusionNet relative to these transfer learning architectures. In addition, an ablation study was performed to specifically assess the contribution of metadata to model performance. The evaluation included four key performance metrics, including classification accuracy, precision, recall, and F1-score. 2. Methods 2.1 Dataset Acquisition The study utilised a secondary dataset compiled from three prior studies: a study on TB diagnosis, the TB cohort study, and the TB Contact and LAM study, all conducted at Epicentre, Mbarara Regional Referral Hospital, Mbarara University of Science of Science and Technology. The dataset comprised 1000 labeled DICOM files of postero-anterior chest radiograph from children under 15 years, including 575 images from children without TB and 425 from children diagnosed with TB. The dataset was split into 70% training, 15% testing, and 15% validation sets. To enhance model generalizability and mitigate overfitting, k-fold cross-validation was applied (Abou Ali et al., 2025 ), and data augmentation techniques, including rotation, scaling, horizontal flips, and intensity variations were performed on the training set (Alomar et al., 2023 ). These strategies collectively enabled a more reliable assessment of the model’s performance on unseen data. 2.2 Metadata and Image Preprocessing Relevant DICOM metadata, including numeric and categorical fields, was extracted while excluding descriptive text tags such as StudyDescription (0008, 1030) and SeriesDescription (0008, 103E) to prevent data leakage. Numeric fields were normalised to [0,1], categorical fields were one-hot encoded, and all features were concatenated into a 256-dimensional numerical tensor for multimodal integration with CNN-derived image features. Chest radiographs were standardized using reproducible preprocessing steps. Pixel intensities were windowed with a level of 50 HU (Hounsfield Unites) and a width of 350 HU, enhancing pulmonary structure visibility. The non-lung regions were masked using automated thresholding morphological operations. Images were resized to 244 x 244 pixel and normalized to [0,1] prior to CNN input. 2.3 Modal Architecture and Training The evaluation included the DICOMFusionNet, alongside conventional CNN-based transfer learning models including Inception V3, VGG16, VGG19, and ResNet. For the DICOMFusionNet, the image branch consisted of convolutional layers with batch normalization and ReLu activation, followed by max-pooling and flattening. The metadata branch processed the 256-dimensional numerical tensor derived from curated DICOM fields through fully connected layers, which were concatenated with image feature prior to classification. Transfer learning models required DICOM image to be converted to PNG or JPEG format. Each model was fine-tuned on the dataset with standard preprocessing and resizing to 244 x 244 pixel. All models were trained using the Adam optimizer (learning rate 1e-4), batch size of 32, and categorical cross-entropy loss. Early stopping based on validation loss and dropout in fully connected layers (rate 0.5) were applied to reduce overfitting. Training was performed on a NVIDIA Tesla V100GPU using PyTorch. An ablation study was conducted to assess the contribution of metadata to DICOMFusionNet’s performance. This study compared the DICOMFusionNet trained on pixel-level image data alone with the DICOMFusionNet trained on both pixel data and embedded metadata. This analysis confirmed that the inclusion of metadata significantly improved classification accuracy, demonstrating that predictive performance was driven by clinically meaningful features rather than spurious correlations in the image data. 3. Results The classification performance of DICOMFusionNet and the conventional transfer learning models is summarised in Table 1 . Performance is reported for training, testing, and validation dataset across accuracy, precision, recall, and F1-score. The table also includes results from the ablation study evaluating the contribution of metadata to DICOMFusionNet’s predictive performance. Table 1 Performance comparison results Model Dataset Split Accuracy (%) Precision Recall F1-score DICOMFusionNet (Image + Metadata) Training 94.1 0.93 0.95 0.94 Testing 92.3 0.91 0.92 0.91 Validation 91.7 0.91 0.92 0.91 DICOMFusionNet (Image Only) Training 89.5 0.88 0.90 0.89 Testing 87.8 0.84 0.86 0.85 Validation 86.9 0.85 0.87 0.86 Inception V3 Training 88.2 0.87 0.88 0.87 Testing 86.7 0.85 0.87 0.86 Validation 85.0 0.84 0.85 0.85 VGG16 Training 87.5 0.86 0.87 0.87 Testing 85.4 0.84 0.85 0.85 Validation 84.1 0.83 0.84 0.84 VGG19 Training 88.0 0.87 0.88 0.87 Testing 85.9 0.84 0.86 0.85 Validation 84.7 0.84 0.85 0.85 ResNet50 Training 89.0 0.88 0.89 0.88 Testing 87.1 0.86 0.87 0.86 Validation 86.5 0.85 0.86 0.86 4. Discussion The evaluation of DICOMFusionNet, a multimodal CNN that integrates raw DICOM images with embedded metadata, demonstrated a superior performance in classifying pediatric chest radiographs compared to conventional transfer learning models. This approach addresses the limitation of traditional models that requires conversion from DICOM to standard image formats, a process that often leads to the loss of clinically relevant metadata and reduced diagnostic accuracy. DICOMFusionNet achieved an accuracy of 92.3% and an F1-socre of 0.91 on the test set, outperforming Inception V3 (86.7%), VGG16 (85.4%), VGG19(85.9%) and ResNet (87.1%). These results are consistent with recent studies highlighting the efficacy of multimodal models in medical image classification. For example, Multimodal Chest X-ray Network (MXC-Net), a context-aware multimodal model for chest radiology, demonstrated improved diagnostic accuracy by integrating structured patient metadata with image data (Yang et al., 2025 ). Similarly, Ben Rabah et al. ( 2025 ) reported enhanced classification performance by combining mammography images with clinical metadata. The ablation study revealed a significant performance drop when DICOMFusionNet was trained using image data alone, with accuracy decreasing to 87.85 and F1-score to 0.85. This emphasizes the critical role of metadata in enhancing model performance. Recent studies have emphasized the importance of preserving both image and metadata in DICOM files to avoid classification bias and loss of diagnostic context (Kathiravelu et al., 2021 ; Mabirizi et al., 2025 ). The transfer learning models, while effective, underperformed relative to DICOMFusionNet. This can be attributed to preprocessing step of converting DICM images to standard formats, which discards valuable metadata and may introduce artifacts. Studies have shown that such conversions can lead to information loss and classification bias (Chiang et al., 2021 ; Kathiravelu et al., 2021 ; Miranda-Viana et al., 2023 ). Additionally, fine-tuning strategies in transfer learning models vary in effectiveness; for example, combining linear probing with full fine-tuning has been found to improve performance in certain medical imaging tasks (Davila et al., 2024 ). The superior performance of DICOMFusionNet highlights the potential of multimodal deep learning models in clinical diagnostic. By preserving and utilizing both image and metadata, these models can provide more accurate and context-aware predictions, thereby supporting clinical decision-making (Musinguzi et al., 2025 ; Tang et al., 2025 ). Furthermore, the integration of metadata can reduce algorithmic bias and enhanced generalization across diverse patient population (Kathiravelu et al., 2021 ). 5. Conclusion The study evaluated the performance of DICOMFusionNet, a multimodal convolutional neural network designed to natively process DICOM files by integrating both pixel-level image data and embedded metadata. The results demonstrated that DICOMFusionNet consistently outperformed conventional transfer learning architecture including Inception V3, VGG16, VGG19, and ResNet50 across all evaluation metrics. The inclusion of metadata proved to be critical factor in boosting predictive accuracy, as shown by the significant performance decline observed in the ablation study when metadata was excluded. These findings highlight the importance of preserving the integrity of DICOM files during deep learning workflow, avoiding conversion to alternative image formats that risk discarding clinically relevant metadata. By maintaining both the image content and contextual metadata, DICOMFusionNet produced more accurate and context-aware classifications, ultimately with clinical diagnostic needs. Beyond demonstrating superior performance, this study highlights the broader potential of multimodal deep learning frameworks to reduce algorithmic bias, improve generalization across diverse patient population, and enhance clinical decision support. Declarations Future work Future research should focus on expanding the diversity of dataset to improve the generalizability of multimodal models. Additionally, exploring the integration of other data modalities, such as electronic health records and genomic data, could further enhance model performance. Addressing computational efficiency and scalability will also be crucial for the widespread adoption of these models in clinical settings. Ethics Statement: The study protocol was reviewed and approved by the Mbarara University of Science and Technology Research Ethics Committee (Protocol No. MUST-2024-1656) in accordance with the guidelines of the Uganda National Council for Science and Technology (UNCST) guidelines. The protocol was registered under approval number SIR495ES. Consent to Participate Declarations: The study utilised a secondary dataset obtained from a previous study that adhered to ethical principles, including the rights of voluntary participation. Informed Consent to Publish: Not applicable Author Contribution: Mabirizi Vicent: conceptualisation, data curation, formal analysis, investigation, methodology, project administration, resources, software, validation, visualisation, writing – original draft, writing – review and editing. Kawuma Simon and Wasswa William : formal analysis, resources, supervision, validation, writing – review and editing. Acknowledgments: We extent our appreciation to Epicentre Mbarara Uganda and Mbarara Regional Referral Hospital for providing the dataset used in this study. Conflicts of Interest: The authors declare no conflict of interest Data Availability Statement: The dataset that supports the findings of this study is available from Epicentre Mbarara Uganda and Mbarara Regional Referral Hospital. Restrictions apply to the availability of this dataset, which were used during the ethical clearance for this study. Funding: The authors received no specific fundings for this work. References Abou Ali, M., Charafeddine, J., Dornaika, F., & Arganda-Carreras, I. (2025). Enhancing generalization and mitigating overfitting in deep learning for brain cancer diagnosis from MRI. Applied Magnetic Resonance , 56 (3), 359–394. Alomar, K., Aysel, H. I., & Cai, X. (2023). Data augmentation in classification and segmentation: A survey and new strategies. Journal of Imaging , 9 (2), 46. Ben Rabah, C., Sattar, A., Ibrahim, A., & Serag, A. (2025). A Multimodal Deep Learning Model for the Classification of Breast Cancer Subtypes. Diagnostics , 15 (8), 995. Chiang, C.-H., Weng, C.-L., & Chiu, H.-W. (2021). Automatic classification of medical image modality and anatomical location using convolutional neural network. Plos One , 16 (6), e0253205. Davila, A., Colan, J., & Hasegawa, Y. (2024). Comparison of fine-tuning strategies for transfer learning in medical image classification. Image and Vision Computing , 146 , 105012. Kathiravelu, P., Sharma, P., Sharma, A., Banerjee, I., Trivedi, H., Purkayastha, S., Sinha, P., Cadrin-Chenevert, A., Safdar, N., & Gichoya, J. W. (2021). A DICOM Framework for Machine Learning and Processing Pipelines Against Real-time Radiology Images. Journal of Digital Imaging , 34 (4), 1005–1013. https://doi.org/10.1007/s10278-021-00491-w Kim, M.-J., & Kim, J.-H. (2022). development of convolutional neural network model for classification of cardiomegaly X-ray images. Journal of Mechanics in Medicine and Biology , 22 (08), 2240020. Koonce, B. (2021). ResNet 50. In Convolutional neural networks with swift for tensorflow: image recognition and dataset categorization (pp. 63–72). Springer. Mabirizi, V., Kawuma, S., Natumanya, D., & Wasswa, W. (2025). Deep Learning Techniques in DICOM Files Classification: A Systematic Review. Artificial Intelligence and Applications . Mayer, R. S., Fliedner, F., Mathisen, I. F., Laib, A., Bein, J., Eichelberg, M., Wild, P. J., & Flinner, N. (2025). Lossy DICOM conversion may affect AI performance. Scientific Reports , 1–10. Meena, G., Mohbey, K. K., Kumar, S., Chawda, R. K., & Gaikwad, S. V. (2023). Image-based sentiment analysis using InceptionV3 transfer learning approach. SN Computer Science , 4 (3), 242. Miranda-Viana, M., Fontenele, R. C., Nogueira-Reis, F., Farias-Gomes, A., Oliveira, M. L., Freitas, D. Q., & Haiter-Neto, F. (2023). DICOM file format has better radiographic image quality than other file formats: an objective study. Brazilian Dental Journal , 34 (4), 150–157. Musinguzi, D., Katumba, A., Murindanyi, S., Nakatumba-Nabende, J., Mwikirize, C., Mbabazi, M., Kisembo, H., Malumba, R., & G Kawooya, M. (2025). PaliGemma-CXR: a multi-task multimodal model for tuberculosis chest X-ray interpretation. BMC Artificial Intelligence , 1 (1), 1–10. Tang, J., Yin, X., Lai, J., Luo, K., & Wu, D. (2025). Fusion of X-Ray Images and Clinical Data for a Multimodal Deep Learning Prediction Model of Osteoporosis: Algorithm Development and Validation Study. JMIR Medical Informatics , 13 , e70738. Varma, D. R. (2012). Managing DICOM images: Tips and tricks for the radiologist. Indian Journal of Radiology and Imaging , 22 (1), 4–13. https://doi.org/10.4103/0971-3026.95396 Veni, N., & Manjula, J. (2022). Modified Visual Geometric Group Architecture for MRI Brain Image Classification. Computer Systems Science & Engineering , 42 (2). Vicent, M., William, W., & Simon, K. (2025). A Multimodal Convolutional Neural Network Based Approach for DICOM Files Classification . 1 (1), 1–10. https://doi.org/10.1049/tje2.70107 Yang, L., Wan, Y., & Pan, F. (2025). Enhancing Chest X-ray Diagnosis with a Multimodal Deep Learning Network by Integrating Clinical History to Refine Attention. Journal of Imaging Informatics in Medicine , 1–16. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7727424","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":561341534,"identity":"d541df11-f19d-4d9b-baf0-7888c19341b1","order_by":0,"name":"Vicent Mabirizi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA10lEQVRIiWNgGAWjYHACNoYEBgkGNvY2BmYStfAcI0ULGEikEanF4NrhZw8e7rGw55N8lvi5gMFOnkEi/QJ+LbfTzA0Snkkws0mnHZaewZBs2CCRU4BXi+TsBDOJhAMSbGzS6Q3SPAzMQH/lJBDQkv4NpIWHTfJ4828ehnrCWvilc8C2SLBJsB0D2nIYqCX9ACEt5QZALQZsPGlp1jMMjhu28bzBq4MB6IVtD38cqLOXbz9mfLugolqenz39AX49qMAAFE08BqRoAQN2kmwZBaNgFIyC4Q8A9Sg48nBYzZ8AAAAASUVORK5CYII=","orcid":"","institution":"Kabale University","correspondingAuthor":true,"prefix":"","firstName":"Vicent","middleName":"","lastName":"Mabirizi","suffix":""},{"id":561341535,"identity":"316f90b8-ef21-4642-bbd3-4dd7ed570076","order_by":1,"name":"Simon Kawuma","email":"","orcid":"","institution":"Mbarara University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Simon","middleName":"","lastName":"Kawuma","suffix":""},{"id":561341536,"identity":"ceb83df9-20ca-4725-a2c5-1eb80951653e","order_by":2,"name":"William Wasswa","email":"","orcid":"","institution":"Mbarara University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"William","middleName":"","lastName":"Wasswa","suffix":""}],"badges":[],"createdAt":"2025-09-27 09:23:13","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7727424/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7727424/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":98625045,"identity":"bd968439-3b94-448c-88e8-f182d0d08d18","added_by":"auto","created_at":"2025-12-19 17:08:54","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":43860,"visible":true,"origin":"","legend":"","description":"","filename":"Manuscriptversiontwo.docx","url":"https://assets-eu.researchsquare.com/files/rs-7727424/v1/c85ea29ba0ecfd8bf78d5fa6.docx"},{"id":98625970,"identity":"6e20ff06-afb5-491e-986e-e05e48f14b2b","added_by":"auto","created_at":"2025-12-19 17:09:27","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5770,"visible":true,"origin":"","legend":"","description":"","filename":"40851140f96e409c942cf1b4126a7c5b.json","url":"https://assets-eu.researchsquare.com/files/rs-7727424/v1/de5a366f0218fe40d0e0e5ac.json"},{"id":98515240,"identity":"fff76f91-9e32-4cfd-9bc3-cf0bcfd2c3e0","added_by":"auto","created_at":"2025-12-18 12:31:57","extension":"xml","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":53465,"visible":true,"origin":"","legend":"","description":"","filename":"40851140f96e409c942cf1b4126a7c5b1enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-7727424/v1/3207490aaa75ffe0fd0db7ef.xml"},{"id":98625036,"identity":"03a018fc-9c01-4384-ab37-104a74a48da3","added_by":"auto","created_at":"2025-12-19 17:08:54","extension":"xml","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":53025,"visible":true,"origin":"","legend":"","description":"","filename":"40851140f96e409c942cf1b4126a7c5b1structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-7727424/v1/16fd429af0fda2fb1485e159.xml"},{"id":98515241,"identity":"ce43a539-8538-45f5-bbfb-c5d29604e8b4","added_by":"auto","created_at":"2025-12-18 12:31:57","extension":"html","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":57043,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-7727424/v1/ca096fa47ef83f054a88877b.html"},{"id":99308781,"identity":"2a232121-a9c9-426e-b854-5562beb502b3","added_by":"auto","created_at":"2025-12-31 16:09:08","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":494896,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7727424/v1/c1a29807-5d3f-450f-ac52-48a285422781.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Evaluation of a Multimodal Convolutional Neural Network-Based Approach for DICOM Files Classification","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eThis paper presents an evaluation of a deep learning model for Digital Imaging and Communication in Medicine (DICOM) image classification, previously developed in our earlier work(Vicent et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2025\u003c/span\u003e), hereafter referred to as DICOMFusionNet. The model was developed to natively process DICOM files using a multimodal Convolutional Neural Network (CNN) architecture that integrates both pixel-level image data and embedded metadata. This approach enhances classification accuracy and reduces bias by eliminating the need for format conversion from DICOM to other formats such as PNG and JPEG, a process that can alter image content or strip ancillary information, thereby affecting downstream model performance (Mayer et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Varma, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2012\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe original study achieved improved results on two datasets including chest X-ray for tuberculosis detection and magnetic resonance imaging (MRI) scans for brain tumor classification. However, although DICOMFusionNet outperformed conventional approaches in internal evaluations, questions remain about its comparative performance and efficiency relative to CCN-based transferring models, including Inception V3 (Meena et al., \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), the Vision Geometry Group (VGG) networks (Veni \u0026amp; Manjula, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2022\u003c/span\u003e), and ResNet50 (Koonce, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2021\u003c/span\u003e), typically require a conversion pipeline from DICOM to formats like PNG or JPEG (Kim \u0026amp; Kim, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Mabirizi et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2025\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eTo answer this question, we conducted a comparative experiment to evaluate the performance of the DICOMFusionNet relative to these transfer learning architectures. In addition, an ablation study was performed to specifically assess the contribution of metadata to model performance. The evaluation included four key performance metrics, including classification accuracy, precision, recall, and F1-score.\u003c/p\u003e"},{"header":"2. Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Dataset Acquisition\u003c/h2\u003e \u003cp\u003eThe study utilised a secondary dataset compiled from three prior studies: a study on TB diagnosis, the TB cohort study, and the TB Contact and LAM study, all conducted at Epicentre, Mbarara Regional Referral Hospital, Mbarara University of Science of Science and Technology. The dataset comprised 1000 labeled DICOM files of postero-anterior chest radiograph from children under 15 years, including 575 images from children without TB and 425 from children diagnosed with TB.\u003c/p\u003e \u003cp\u003eThe dataset was split into 70% training, 15% testing, and 15% validation sets. To enhance model generalizability and mitigate overfitting, k-fold cross-validation was applied (Abou Ali et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2025\u003c/span\u003e), and data augmentation techniques, including rotation, scaling, horizontal flips, and intensity variations were performed on the training set (Alomar et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). These strategies collectively enabled a more reliable assessment of the model\u0026rsquo;s performance on unseen data.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Metadata and Image Preprocessing\u003c/h2\u003e \u003cp\u003eRelevant DICOM metadata, including numeric and categorical fields, was extracted while excluding descriptive text tags such as StudyDescription (0008, 1030) and SeriesDescription (0008, 103E) to prevent data leakage. Numeric fields were normalised to [0,1], categorical fields were one-hot encoded, and all features were concatenated into a 256-dimensional numerical tensor for multimodal integration with CNN-derived image features.\u003c/p\u003e \u003cp\u003eChest radiographs were standardized using reproducible preprocessing steps. Pixel intensities were windowed with a level of 50 HU (Hounsfield Unites) and a width of 350 HU, enhancing pulmonary structure visibility. The non-lung regions were masked using automated thresholding morphological operations. Images were resized to 244 x 244 pixel and normalized to [0,1] prior to CNN input.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Modal Architecture and Training\u003c/h2\u003e \u003cp\u003eThe evaluation included the DICOMFusionNet, alongside conventional CNN-based transfer learning models including Inception V3, VGG16, VGG19, and ResNet.\u003c/p\u003e \u003cp\u003eFor the DICOMFusionNet, the image branch consisted of convolutional layers with batch normalization and ReLu activation, followed by max-pooling and flattening. The metadata branch processed the 256-dimensional numerical tensor derived from curated DICOM fields through fully connected layers, which were concatenated with image feature prior to classification.\u003c/p\u003e \u003cp\u003eTransfer learning models required DICOM image to be converted to PNG or JPEG format. Each model was fine-tuned on the dataset with standard preprocessing and resizing to 244 x 244 pixel. All models were trained using the Adam optimizer (learning rate 1e-4), batch size of 32, and categorical cross-entropy loss. Early stopping based on validation loss and dropout in fully connected layers (rate 0.5) were applied to reduce overfitting. Training was performed on a NVIDIA Tesla V100GPU using PyTorch.\u003c/p\u003e \u003cp\u003eAn ablation study was conducted to assess the contribution of metadata to DICOMFusionNet\u0026rsquo;s performance. This study compared the DICOMFusionNet trained on pixel-level image data alone with the DICOMFusionNet trained on both pixel data and embedded metadata. This analysis confirmed that the inclusion of metadata significantly improved classification accuracy, demonstrating that predictive performance was driven by clinically meaningful features rather than spurious correlations in the image data.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. Results","content":"\u003cp\u003eThe classification performance of DICOMFusionNet and the conventional transfer learning models is summarised in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. Performance is reported for training, testing, and validation dataset across accuracy, precision, recall, and F1-score. The table also includes results from the ablation study evaluating the contribution of metadata to DICOMFusionNet\u0026rsquo;s predictive performance.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePerformance comparison results\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDataset Split\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePrecision\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eF1-score\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eDICOMFusionNet\u003c/p\u003e \u003cp\u003e(Image\u0026thinsp;+\u0026thinsp;Metadata)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTraining\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e94.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.95\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.94\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTesting\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e92.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.91\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.92\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.91\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eValidation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e91.7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.91\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.92\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.91\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eDICOMFusionNet\u003c/p\u003e \u003cp\u003e(Image Only)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTraining\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e89.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.88\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.90\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.89\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTesting\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e87.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.85\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eValidation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e86.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.85\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.86\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eInception V3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTraining\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e88.2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.88\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.87\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTesting\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e86.7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.85\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.86\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eValidation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e85.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.85\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.85\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eVGG16\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTraining\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e87.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.87\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTesting\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e85.4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.85\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.85\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eValidation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e84.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.83\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.84\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eVGG19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTraining\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e88.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.88\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.87\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTesting\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e85.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.85\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eValidation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e84.7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.85\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.85\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eResNet50\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTraining\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e89.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.88\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.89\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.88\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTesting\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e87.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.86\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eValidation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e86.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.85\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.86\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e"},{"header":"4. Discussion","content":"\u003cp\u003eThe evaluation of DICOMFusionNet, a multimodal CNN that integrates raw DICOM images with embedded metadata, demonstrated a superior performance in classifying pediatric chest radiographs compared to conventional transfer learning models. This approach addresses the limitation of traditional models that requires conversion from DICOM to standard image formats, a process that often leads to the loss of clinically relevant metadata and reduced diagnostic accuracy.\u003c/p\u003e \u003cp\u003eDICOMFusionNet achieved an accuracy of 92.3% and an F1-socre of 0.91 on the test set, outperforming Inception V3 (86.7%), VGG16 (85.4%), VGG19(85.9%) and ResNet (87.1%). These results are consistent with recent studies highlighting the efficacy of multimodal models in medical image classification. For example, Multimodal Chest X-ray Network (MXC-Net), a context-aware multimodal model for chest radiology, demonstrated improved diagnostic accuracy by integrating structured patient metadata with image data (Yang et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). Similarly, Ben Rabah et al. (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) reported enhanced classification performance by combining mammography images with clinical metadata.\u003c/p\u003e \u003cp\u003eThe ablation study revealed a significant performance drop when DICOMFusionNet was trained using image data alone, with accuracy decreasing to 87.85 and F1-score to 0.85. This emphasizes the critical role of metadata in enhancing model performance. Recent studies have emphasized the importance of preserving both image and metadata in DICOM files to avoid classification bias and loss of diagnostic context (Kathiravelu et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Mabirizi et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2025\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe transfer learning models, while effective, underperformed relative to DICOMFusionNet. This can be attributed to preprocessing step of converting DICM images to standard formats, which discards valuable metadata and may introduce artifacts. Studies have shown that such conversions can lead to information loss and classification bias (Chiang et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Kathiravelu et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Miranda-Viana et al., \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Additionally, fine-tuning strategies in transfer learning models vary in effectiveness; for example, combining linear probing with full fine-tuning has been found to improve performance in certain medical imaging tasks (Davila et al., \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe superior performance of DICOMFusionNet highlights the potential of multimodal deep learning models in clinical diagnostic. By preserving and utilizing both image and metadata, these models can provide more accurate and context-aware predictions, thereby supporting clinical decision-making (Musinguzi et al., \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Tang et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). Furthermore, the integration of metadata can reduce algorithmic bias and enhanced generalization across diverse patient population (Kathiravelu et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2021\u003c/span\u003e).\u003c/p\u003e"},{"header":"5. Conclusion","content":"\u003cp\u003eThe study evaluated the performance of DICOMFusionNet, a multimodal convolutional neural network designed to natively process DICOM files by integrating both pixel-level image data and embedded metadata. The results demonstrated that DICOMFusionNet consistently outperformed conventional transfer learning architecture including Inception V3, VGG16, VGG19, and ResNet50 across all evaluation metrics. The inclusion of metadata proved to be critical factor in boosting predictive accuracy, as shown by the significant performance decline observed in the ablation study when metadata was excluded.\u003c/p\u003e \u003cp\u003eThese findings highlight the importance of preserving the integrity of DICOM files during deep learning workflow, avoiding conversion to alternative image formats that risk discarding clinically relevant metadata. By maintaining both the image content and contextual metadata, DICOMFusionNet produced more accurate and context-aware classifications, ultimately with clinical diagnostic needs.\u003c/p\u003e \u003cp\u003eBeyond demonstrating superior performance, this study highlights the broader potential of multimodal deep learning frameworks to reduce algorithmic bias, improve generalization across diverse patient population, and enhance clinical decision support.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eFuture work\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFuture research should focus on expanding the diversity of dataset to improve the generalizability of multimodal models. Additionally, exploring the integration of other data modalities, such as electronic health records and genomic data, could further enhance model performance. Addressing computational efficiency and scalability will also be crucial for the widespread adoption of these models in clinical settings.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEthics Statement:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe study protocol was reviewed and approved by the Mbarara University of Science and Technology Research Ethics Committee (Protocol No. MUST-2024-1656) in accordance with the guidelines of the Uganda National Council for Science and Technology (UNCST) guidelines. The protocol was registered under approval number SIR495ES.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent to Participate Declarations:\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe study utilised a secondary dataset obtained from a previous study that adhered to ethical principles, including the rights of voluntary participation.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eInformed Consent to Publish:\u0026nbsp;\u003c/strong\u003eNot applicable\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor Contribution:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMabirizi Vicent:\u003c/strong\u003e conceptualisation, data curation, formal analysis, investigation, methodology, project administration, resources, software, validation, visualisation, writing \u0026ndash; original draft, writing \u0026ndash; review and editing. \u003cstrong\u003eKawuma Simon\u003c/strong\u003e and \u003cstrong\u003eWasswa William\u003c/strong\u003e: formal analysis, resources, supervision, validation, writing \u0026ndash; review and editing.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgments:\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe extent our appreciation to Epicentre Mbarara Uganda and Mbarara Regional Referral Hospital for providing the dataset used in this study.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConflicts of Interest:\u0026nbsp;\u003c/strong\u003eThe authors declare no conflict of interest\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData Availability Statement:\u0026nbsp;\u003c/strong\u003eThe dataset that supports the findings of this study is available from Epicentre Mbarara Uganda and Mbarara Regional Referral Hospital. Restrictions apply to the availability of this dataset, which were used during the ethical clearance for this study. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding:\u0026nbsp;\u003c/strong\u003eThe authors received no specific fundings for this work.\u0026nbsp;\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAbou Ali, M., Charafeddine, J., Dornaika, F., \u0026amp; Arganda-Carreras, I. (2025). Enhancing generalization and mitigating overfitting in deep learning for brain cancer diagnosis from MRI. \u003cem\u003eApplied Magnetic Resonance\u003c/em\u003e, \u003cem\u003e56\u003c/em\u003e(3), 359\u0026ndash;394.\u003c/li\u003e\n\u003cli\u003eAlomar, K., Aysel, H. I., \u0026amp; Cai, X. (2023). Data augmentation in classification and segmentation: A survey and new strategies. \u003cem\u003eJournal of Imaging\u003c/em\u003e, \u003cem\u003e9\u003c/em\u003e(2), 46.\u003c/li\u003e\n\u003cli\u003eBen Rabah, C., Sattar, A., Ibrahim, A., \u0026amp; Serag, A. (2025). A Multimodal Deep Learning Model for the Classification of Breast Cancer Subtypes. \u003cem\u003eDiagnostics\u003c/em\u003e, \u003cem\u003e15\u003c/em\u003e(8), 995.\u003c/li\u003e\n\u003cli\u003eChiang, C.-H., Weng, C.-L., \u0026amp; Chiu, H.-W. (2021). Automatic classification of medical image modality and anatomical location using convolutional neural network. \u003cem\u003ePlos One\u003c/em\u003e, \u003cem\u003e16\u003c/em\u003e(6), e0253205.\u003c/li\u003e\n\u003cli\u003eDavila, A., Colan, J., \u0026amp; Hasegawa, Y. (2024). Comparison of fine-tuning strategies for transfer learning in medical image classification. \u003cem\u003eImage and Vision Computing\u003c/em\u003e, \u003cem\u003e146\u003c/em\u003e, 105012.\u003c/li\u003e\n\u003cli\u003eKathiravelu, P., Sharma, P., Sharma, A., Banerjee, I., Trivedi, H., Purkayastha, S., Sinha, P., Cadrin-Chenevert, A., Safdar, N., \u0026amp; Gichoya, J. W. (2021). A DICOM Framework for Machine Learning and Processing Pipelines Against Real-time Radiology Images. \u003cem\u003eJournal of Digital Imaging\u003c/em\u003e, \u003cem\u003e34\u003c/em\u003e(4), 1005\u0026ndash;1013. https://doi.org/10.1007/s10278-021-00491-w\u003c/li\u003e\n\u003cli\u003eKim, M.-J., \u0026amp; Kim, J.-H. (2022). development of convolutional neural network model for classification of cardiomegaly X-ray images. \u003cem\u003eJournal of Mechanics in Medicine and Biology\u003c/em\u003e, \u003cem\u003e22\u003c/em\u003e(08), 2240020.\u003c/li\u003e\n\u003cli\u003eKoonce, B. (2021). ResNet 50. In \u003cem\u003eConvolutional neural networks with swift for tensorflow: image recognition and dataset categorization\u003c/em\u003e (pp. 63\u0026ndash;72). Springer.\u003c/li\u003e\n\u003cli\u003eMabirizi, V., Kawuma, S., Natumanya, D., \u0026amp; Wasswa, W. (2025). Deep Learning Techniques in DICOM Files Classification: A Systematic Review. \u003cem\u003eArtificial Intelligence and Applications\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003eMayer, R. S., Fliedner, F., Mathisen, I. F., Laib, A., Bein, J., Eichelberg, M., Wild, P. J., \u0026amp; Flinner, N. (2025). Lossy DICOM conversion may affect AI performance. \u003cem\u003eScientific Reports\u003c/em\u003e, 1\u0026ndash;10.\u003c/li\u003e\n\u003cli\u003eMeena, G., Mohbey, K. K., Kumar, S., Chawda, R. K., \u0026amp; Gaikwad, S. V. (2023). Image-based sentiment analysis using InceptionV3 transfer learning approach. \u003cem\u003eSN Computer Science\u003c/em\u003e, \u003cem\u003e4\u003c/em\u003e(3), 242.\u003c/li\u003e\n\u003cli\u003eMiranda-Viana, M., Fontenele, R. C., Nogueira-Reis, F., Farias-Gomes, A., Oliveira, M. L., Freitas, D. Q., \u0026amp; Haiter-Neto, F. (2023). DICOM file format has better radiographic image quality than other file formats: an objective study. \u003cem\u003eBrazilian Dental Journal\u003c/em\u003e, \u003cem\u003e34\u003c/em\u003e(4), 150\u0026ndash;157.\u003c/li\u003e\n\u003cli\u003eMusinguzi, D., Katumba, A., Murindanyi, S., Nakatumba-Nabende, J., Mwikirize, C., Mbabazi, M., Kisembo, H., Malumba, R., \u0026amp; G Kawooya, M. (2025). PaliGemma-CXR: a multi-task multimodal model for tuberculosis chest X-ray interpretation. \u003cem\u003eBMC Artificial Intelligence\u003c/em\u003e, \u003cem\u003e1\u003c/em\u003e(1), 1\u0026ndash;10.\u003c/li\u003e\n\u003cli\u003eTang, J., Yin, X., Lai, J., Luo, K., \u0026amp; Wu, D. (2025). Fusion of X-Ray Images and Clinical Data for a Multimodal Deep Learning Prediction Model of Osteoporosis: Algorithm Development and Validation Study. \u003cem\u003eJMIR Medical Informatics\u003c/em\u003e, \u003cem\u003e13\u003c/em\u003e, e70738.\u003c/li\u003e\n\u003cli\u003eVarma, D. R. (2012). Managing DICOM images: Tips and tricks for the radiologist. \u003cem\u003eIndian Journal of Radiology and Imaging\u003c/em\u003e, \u003cem\u003e22\u003c/em\u003e(1), 4\u0026ndash;13. https://doi.org/10.4103/0971-3026.95396\u003c/li\u003e\n\u003cli\u003eVeni, N., \u0026amp; Manjula, J. (2022). Modified Visual Geometric Group Architecture for MRI Brain Image Classification. \u003cem\u003eComputer Systems Science \u0026amp; Engineering\u003c/em\u003e, \u003cem\u003e42\u003c/em\u003e(2).\u003c/li\u003e\n\u003cli\u003eVicent, M., William, W., \u0026amp; Simon, K. (2025). \u003cem\u003eA Multimodal Convolutional Neural Network Based Approach for DICOM Files Classification\u003c/em\u003e. \u003cem\u003e1\u003c/em\u003e(1), 1\u0026ndash;10. https://doi.org/10.1049/tje2.70107\u003c/li\u003e\n\u003cli\u003eYang, L., Wan, Y., \u0026amp; Pan, F. (2025). Enhancing Chest X-ray Diagnosis with a Multimodal Deep Learning Network by Integrating Clinical History to Refine Attention. \u003cem\u003eJournal of Imaging Informatics in Medicine\u003c/em\u003e, 1\u0026ndash;16.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"DICOMFusionNet, Multimodal Deep Learning, Medical Image Classification, Metadata Integration","lastPublishedDoi":"10.21203/rs.3.rs-7727424/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7727424/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe Digital Imaging and Communication in Medicine (DICOM) standard preserve both pixel-level image data and clinically relevant metadata. However, conventional deep learning pipelines for medical image classification often discard this metadata by converting DICOM files into formats such as PNG or JPEG, leading to information loss and potential bias. This study evaluated the performance of DICOMFusionNet, a multimodal convolutional neural network (CNN) developed to natively process DICOM files by integrating both image data and embedded metadata, in comparison with widely used transfer learning models. A dataset of 1000 pediatric chest radiographs (425 tuberculosis-positive and 575 controls) from Epicentre, Mbarara Regional Referral Hospital, was used. Images were pre-processed to enhance pulmonary visibility, and relevant metadata fields were normalized and one-hot encoded for integration. DICOMFusionNet was benchmarked against Inception V3, VGG16, VGG19 and ResNet50, all requiring DICOM-to-PNG conversion. Performance was evaluated using accuracy, precision, recall, and F1-score. An ablation study assessed the contribution of metadata to classification performance. DICOMFusionNet achieved superior performance with a test accuracy of 92.3% and F1-score of 0.91, outperforming Inception V3 (86.7%), VGG16 (85.4%), VGG19 (85.9%), and ResNet50 (87.1%). The ablation study revealed a significant drop in accuracy (87.8%) and F1-score (0.85) when metadata was excluded, highlighting its critical role in predictive performance. DICOMFusionNet demonstrates that preserving both image and metadata in medical imaging tasks yields more accurate and context-aware classification. This multimodal approach reduces bias, enhance generalization, and provides a promising framework for clinical decision support in diagnostic imaging.\u003c/p\u003e","manuscriptTitle":"Evaluation of a Multimodal Convolutional Neural Network-Based Approach for DICOM Files Classification","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-18 12:31:53","doi":"10.21203/rs.3.rs-7727424/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"85338f0f-73a9-480b-8090-fd824409136e","owner":[],"postedDate":"December 18th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-12-23T10:09:08+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-18 12:31:53","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7727424","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7727424","identity":"rs-7727424","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}