Applying Multimodal Data Fusion based on Deep Learning Methods for the Diagnosis of Neglected Tropical Diseases: A Systematic Review

Applying Multimodal Data Fusion based on Deep Learning Methods for the Diagnosis of Neglected Tropical Diseases: A Systematic Review | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Applying Multimodal Data Fusion based on Deep Learning Methods for the Diagnosis of Neglected Tropical Diseases: A Systematic Review Yohannes Minyilu, Mohammed Abebe, Million Meshesha This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3870993/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Neglected tropical diseases (NTDs) are the most prevalent diseases worldwide, affecting one-tenth of the world population. Although there are multiple approaches to diagnosing these diseases, using skin manifestations and lesions caused as a result of these diseases along with other medical records is the preferred method. This fact triggers the need to explore and implement a deep learning-based diagnostic model using multimodal data fusion (MMDF) techniques to enhance the diagnostic process. This paper, thus, endeavors to present a thorough systematic review of studies regarding the implementation of MMDF techniques for the diagnosis of skin-related NTDs. To achieve its objective, the study used the PRISMA method based on predefined questions and collected 427 articles from seven major and reputed sources and critically appraised each article. Since no previous studies were found regarding the implementation of MMDF for the diagnoses of skin related NTDs, similar studies using MMDF for the diagnoses of other skin diseases, such as skin cancer, were collected and analyzed in this review to extract information about the implementation of these methods. In doing so, various studies are analyzed using six different parameters, including research approaches, disease selected for diagnosis, dataset, algorithms, performance achievements, and future directions. Accordingly, although all the studies used diverse research methods and datasets based on their problems, deep learning-based convolutional neural networks (CNN) algorithms are found to be the most frequently used and best-performing models in all the studies reviewed. Deep Learning Disease Diagnosis Multimodal Data Fusion Skin NTDs Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 1. Introduction Being the largest organ in the human body, the skin can serve as an indicator of some illnesses arising from different causes such as cancer, internal organ failure, and neglected tropical diseases. NTDs are the most prevalent diseases globally affecting more than one billion people worldwide (i.e., more than ten percent of the world’s population), particularly, in the tropical areas of the world among the poorest, most vulnerable and outcast groups and still have devastating impacts on people's physical, mental, and social well-being [ 1 ][ 2 ][ 3 ][ 4 ]. However, these diseases can be diagnosed using skin related symptoms since majority of the NTDs have primary skin indicators or associated clinical features where 18 of the 20 NTDs (recognized by the World Health Organization (WHO)) having skin related symptoms [ 5 ]. Hence, the utilization of DL-based diagnostic systems for the diagnoses and recognition of skin related NTDs will be a great achievement in overcoming the NTDs. This study endeavored to present a thorough systematic review of studies regarding the implementation of MMDF techniques for the diagnoses of skin related NTDs. Since no previous studies implemented MMDF techniques for the diagnoses of skin NTDs, related studies conducted for the diagnoses of skin diseases other than NTDs using MMDF and DL methods were deeply appraised by this review. These studies confirmed that the utilization of MMDF techniques outperforms the traditional diagnostic models that implemented DL methods without MMDF [ 6 ][ 7 ][ 8 ][ 9 ]. It is in view of these facts the study is -motivated to conduct this systematic literature review and a thorough appraisal of previous studies using the PRISMA method of systematic review based on the following guiding questions: What DL methods or approaches were utilized for the diagnoses of the skin disease(s)? Which data fusion methods were used for the skin disease diagnosis tasks? What types of medical data were integrated to demonstrate MMDF method for the diagnoses of the skin diseases? Which algorithms were used and how does each algorithm perform in the DL-based MMDF skin disease diagnostic model or system? 2. The Need for Intelligent Diagnostic Systems In recent times, due to the high desire to enhance the diagnostic processes in the healthcare sectors, the utilization of automated and intelligent diagnostic systems is getting greater attention for the diagnosis of various diseases. In this regard, intelligent diagnostic systems built based on machine learning (ML) and deep learning (DL) methods are the most researched and deployed approaches in the healthcare sector to support diagnostic decision making. On the other hand, in the real-world clinical settings, efficient disease diagnostic processes are basically carried out by using different clinical data that are taken from different sources and different formats or modalities including textual patient information and medical clinical images such as X-ray, dermoscopic images or even patient skin images. The integrative utilization of the diverse modalities of medical data can be used to enhance the diagnostic processes, thereby enhancing the quality of healthcare services, by using the ML and DL methods. In ML, this process of integrating multiple modalities of data (possibly taken from different sources) is technically called multimodal data fusion [ 10 ][ 11 ]. Multimodal data techniques are playing vital roles in developing intelligent disease diagnostic systems for different diseases such as in dermatology [ 12 ]. In this regard, MMDF techniques are advancing diagnostic accuracy, where these methods outperform other baseline methods, as presented in [ 13 ]. 2.1. Deep Learning and Diagnoses of NTDs The current diagnostic approaches used for NTDs are mainly based on clinical procedures, such as patient observation and laboratory examinations based on limited resources in most affected areas. Currently, however, there are efforts towards the utilization of intelligent diagnostic tools using ML and DL approaches. Since most of the NTDs are curtly being diagnosed using skin manifestation, the utilization of DL-based approaches for the diagnoses of these diseases would be a great potential to support and enhance the diagnostic processes. In this regard, different studies were previously conducted to diagnose various NTDs. (Beesetty et al. 2023) [ 14 ], conducted a study towards leprosy skin lesion detection employing a Siamese (Siamese NN)-based few-shot learning (FSL) model for a small clinical dataset and claimed a higher diagnostic accuracy. On the other hand, (Ali et al. 2022) [ 15 ], used ML methods for early prediction of Schistosomiasis and concluded with the CatBoost model showing the best performance with the highest accuracy being above 80%. An optimized diagnostic approach was also proposed for NTDs by selecting three diseases and developing a model using SVM and the black hole algorithm (BHO) achieving more than 90% accuracy [ 16 ]. Another study reviewed by the current study demonstrated a DL-based diagnostic model for NTDs using skin images only and achieved 70% accuracy [ 17 ]. All the aforementioned studies utilized ML and DL methods for the diagnoses of NTDs and achieved remarkable results in terms of accuracy. However, no previous studies were found that utilized DL-based methods using MMDF techniques which will help to achieve higher diagnostic accuracies, as experimented in other studies for non-NTD skin diseases which require further research. 2.2. Data Fusion Approaches Data or information fusion represents the usage of data or information from different sources in different formats or modalities for interpretation in all tasks that require any type of parameter estimation or prediction using data or information [ 18 ]. There are different fusion techniques to combine and aggregate multimodal data which include feature-level fusion, decision-level or late fusion, hybrid multimodal fusion, model-level fusion, rule-based fusion, classification-based fusion and estimation-based fusion [ 19 ]. 2.2.1. Feature Fusion Feature fusion is a data integration technique used to aggregate multiple feature sets extracted from multiple input data to generate a single feature set [ 19 ]. In image processing problems, it refers to the fusion of feature vectors of training images extracted from shared weight network layer and feature vectors composed of other numerical data [ 20 ]. It helps to learn image features fully for the description of their rich internal information [ 21 ]. Various studies are found and appraised that use feature fusion techniques to develop diagnostic models for the diagnoses of skin diseases. 2.2.2. Model Fusion Model fusion, also known as late fusion, represents a fusion approach that combines different models. The study done by (AlDahoul et al 2021), [ 22 ], combines two deep neural networks including binary normal/attack classifier and multi-attack classifier to train a deep neural network (DNN) for network anomaly detection. As mentioned by (Shoumy et al 2020) [ 19 ], the model fusion technique uses the connection between experimental data under different modalities. 2.2.3. Image Fusion Image fusion combines different images and generates informative images by integrating images obtained from different sources [ 23 ]. A previous study by (Y. Wang et al 2021), [ 24 ], suggested that aggregating medical images helps to enhance diagnostic accuracy. This claim was demonstrated by fusing clinical images and dermoscopic images using the deep convolutional neural networks (DCNN) methods and achieved an overall accuracy above 80%. While the clinical images are clinically captured photographs [ 25 ], dermoscopic images represent images taken by dermatologists using dermoscopy [ 26 ]. 2.2.4. Multimodal Data Fusion Multimodal data represents the different formats or modalities of data such as text, image, video, and audio. A multimodal data fusion approach is used for combining particular modalities to derive multimodal representation [ 10 ][ 11 ][ 19 ][ 27 ]. This approach has multiple applications for healthcare systems as it allows the combination of different modalities of data, such as textual medical history of patients, clinical images of patients (such as skin images of patients) to form a single multimodal dataset that can be used to train diagnostic models using ML and DL methods. In this regard, various studies implemented and demonstrated MMDF for the diagnoses of different skin diseases as summarized in Table 3, Appendix . 3. Materials and Methods To conduct and report this systematic review, we follow as a basis, the steps suggested by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) model [ 28 ][ 29 ] as described below. 3.1. Search Strategy Research articles were searched from all the major indexing databases and search engines, such as Google Scholar, PubMed, IEEE Explore, and ScienceDirect. In addition, proper search keywords were prepared and used while searching for the articles, where the search keywords have the appropriate level of relationships with the topics and contents of the articles. A set of searching keywords have been used to deeply search and filter the articles. In this regard, Boolean operators “AND” and “OR” were mainly used. The “AND” operator was used to search for articles in a specific research area to narrow down the search results; “multimodal medical data” AND “data fusion” to search for articles containing both phrases. On the other hand, the “OR” Boolean operator was used to search for articles from wider perspectives as this operator broadens the search results such as “machine learning” OR “deep learning”. Using the specified methods and operators, multiple search keywords were initially prepared and used to find a sufficient amount of relevant articles. Although a lot of search keywords were used while searching for the articles, some of the keywords include ["Neglected Tropical Diseases" OR "NTDs" AND "Diagnosis" OR "diagnostic model" AND "Deep Learning" OR "DL" OR "Convolutional Neural Network" OR "CNN" OR "Deep Neural Network" OR "DNN" OR "Recurrent Neural Network" OR "RNN"], ["Neglected Tropical Diseases" OR "NTDs" AND "Diagnosis" OR "diagnostic model" AND "Deep Learning" OR "DL" or "Convolutional Neural Network" OR "CNN" OR "Deep Neural Network" OR "DNN" OR "Recurrent Neural Network" OR "RNN" AND "Data Fusion" OR "Multimodal medical Data" OR "Multimodal Data Fusion"], [(((deep learning) AND ((diagnostic model) OR (diagnostic system) OR (diagnostic tool)) AND (skin diseases) AND (skin images)) AND (medical record)) AND (data fusion))]. 3.2. Eligibility Criteria Not all articles are critically relevant for the review concerning the integration of multimodal data fusion techniques based on DL methods for the diagnosis of skin related NTDs. Hence, a set of inclusion and/or exclusion criteria are applied, as shown below. Articles must demonstrate DL methods for the diagnosis of skin NTDs, at least skin diseases if not implemented for skin NTDs, with proper evaluation of the methods used. Articles must demonstrate proper utilization of multimodal data fusion techniques for the diagnosis of skin NTDs, at least skin diseases since there are no previous studies that use multimodal data fusion techniques for the diagnosis of skin NTDs so far. Articles must incorporate precise presentation or discussion and evaluation of all the methods and techniques used in that particular article. An Article that used DL methods for the diagnosis of skin related diseases other than NTDs is selected if that particular article uses new or emerging DL methods and presents a proper analysis of the methods and techniques used for the diagnosis of that particular skin disease(s). However, articles that use the popular and previously used DL methods for the diagnosis of diseases other than skin diseases have fewer chances to be selected. The article should have an appropriate level of similarity and relationships in its topics and contents with the searching keywords used to deeply search and filter the articles. Articles that do not utilize DL and data fusion techniques are excluded from analysis. Articles published in languages other than English are excluded from analysis. Finally, articles published prior to the year 2014 are also excluded. 3.3. Article Search Different searching methods such as ‘basic search’ and ‘advanced search’ methods were used on multiple article sources. First, the ordinary or basic searching method was used where general titles and the proposed keywords were entered in the regular ‘search box’ of each of the databases and searched. Secondly, the ‘advanced search’ option was used which allows to specify subject areas, related topics, publication dates and other relevant options which helps to obtain articles that are relevant to the topic by narrowing down the search results. Using both of the search methods and search keywords, a thorough and rigorous searching was conducted on multiple search engines, journals, databases and libraries to find relevant articles. The sources include Google Scholar, IEEE Explore, MDPI, Mendeley, Nature, PubMed, ScienceDirect, AJOL, IDP, NCBI, PLOS, Springer, and Tropical Medicine and Health. Finally, by specifying article publication dates and applying the searching methods on the different databases, 427 articles that were published between the year 2014 and 2024 were collected and prepared for screening. Each database was used independently to search articles. Furthermore, previously searched sources such as academic web portals, academic libraries and research sites were also used as there were relevant documents from these sources. Hence, 16 articles were collected from such sources. However, no relevant articles were found related to this particular area. 3.4. Relevant Article Selection To select relevant articles, an extensive searching method was used using wider options of searching keywords. The entire process of article selection for this review was conducted based on the PRISMA method as it was an evidence-based minimum set of items for reporting for systematic reviews and meta-analyses [ 29 ][ 28 ]. This was done to provide insights for the recent and future research regarding the utilization of DL methods for NTDs diagnosis, the integration of data fusion techniques if they have been used for NTDs and finally to assess and present feedback so as to enhance the performances of such DL-based models. To select relevant articles for this review, five levels of screening were performed where the first level screening was conducted manually using file names and titles. Then, the next levels of screenings were performed using software tools such as ‘EndNote’ and ‘Rayyan’. As a reference management tool, EndNote was used to create a library containing the collected articles and for manipulation and data processing to check duplicate files in the library. Finally, regarding the screening process, using a higher-level screening software tool is mandatory, and for this purpose, Rayyan, a free online software tool [ 30 ] was employed which is mainly used to speed up the literature screening process in systematic reviews. This online tool uses the article library exported from EndNote and it was first used to check duplication based on title, author names and abstracts. 4. Results and Analysis A series of screening operations were implemented on the collected articles in order to identify the most relevant set of articles for this review. In this case, the first level screening was conducted manually on a total of 427 files using file names and titles of the article which allowed as collecting 397 items were selected out of the total 427 articles. Using EndNote to create a library resulted in the automatic removal of 25 articles as there were duplicate files from different folders, followed by an automatic duplicate detection, leading to a library containing 371 articles. Further screening using ‘Rayyan’, 4 duplicated articles were detected in the library and two of them were removed where 369 articles were finally identified. Further, using this online software tool, 90 articles that have a relationship with the current topic of the study were selected based on title and abstract analysis. Additional screening was required to identify articles in relation to the study area and 18 articles were identified out of the 90 related articles. Finally, 9 articles were selected for the final analysis. The overall article selection procedure is outlined using the PRISMA flow chart as depicted in Fig. 1 below. 4.1. Distribution of Articles By applying the specified searching methods on the seven different databases, 427 articles that were published between the year 2014 and 2024 were collected as shown in Fig. 2 below. Accordingly, Google Scholar was primarily used and it allowed us to collect 178 articles from the different sources including IEEE Explore, MDPI, Mendeley, Nature, PubMed, ScienceDirect, AJOL, IDP, NCBI, PLOS, Springer, and Tropical Medicine and Health. Furthermore, additional analysis was performed regarding the sources of the articles with respect to the first two consecutive initial levels of screening, as shown in Fig. 3 below. 4.2. Distribution of Articles by Publication Year After conducting three levels of screening, 90 articles that have direct relationship with the current systematic review have been selected for further screening based on full-text reading and analyses. The selected articles and their respective publication year along with the distribution of the publications years have been shown in Fig. 4 below. As shown, the articles used for this systematic review included studies that have been published recently, where the majority of the studies representing 31% are articles published in 2023, 25% were published in 2022, 16% were published in 2021, 14% were published in 2020, and the remaining 14% were articles published from 2014–2019. 4.3. Distribution of Articles by Methods Used Finally, the 90 articles were further analyzed by categorizing them into four different groups, (i) articles that utilized DL methods for the diagnosis of skin diseases, (ii) articles that implement ML & DL techniques for the diagnosis of NTDs, (iii) articles about the implementation of multimodal data fusion techniques for medical data fusion, and (iv), articles that implement multimodal data fusion based on DL-based methods for the diagnosis of skin diseases as shown in Fig. 5 below. As portrayed in Fig. 5 below, 54.44% of articles utilized ML and DL methods for the diagnosis of skin diseases in general, 20% deal with multimodal data fusion techniques for healthcare systems and 20% implementation of DL-based multimodal data fusion methods for the diagnosis of skin diseases. On the other hand, 5.56% of the articles utilized ML and DL methods for the diagnoses of NTDs in general have been identified and analyzed. However, no article has been found that deals with the implementation of DL-based MMDF methods for the diagnosis of NTDs which has led to the analyses of previous studies that used this approach for the diagnoses of different skin diseases other than the NTDs. By conducting the fourth level screening, 18 articles that utilize different fusion techniques for the diagnosis of various skin diseases have been identified. 4.4. Analysis of Fusion Techniques Used The final screening has resulted in the separation of 7 of the 18 articles due to the fusion techniques they utilize for the diagnosis of skin diseases. The fusion techniques presented in those 7 studies are feature fusion (5 studies), image fusion (1 study) and model fusion (1 review study) as presented in Table 1 below. Table 1 presented the analysis of three different types of fusion other than MMDF using five different parameters as shown in the table below. On the other hand, 2 articles presented a review of the multimodal data fusion techniques for the diagnoses of skin diseases other than NTDs. Although the 2 articles [ 12 ][ 13 ], didn’t implement MMDF techniques for a specific skin disease diagnosis using their datasets of preferences, they presented theoretical analyses. All in all, 9 articles are used for the final analysis of this review. Table 1 Review of the future fusion and related techniques for skin disease diagnoses Ref Pub. Yr. Study Method / Approach Used Disease(s) Selected Dataset(s) Used Algorithm(s) Used Performance Results Achieved [ 31 ] 2019 Transfer Learning and multi-layer feature fusion network Skin Lesion HAM10000 dataset CNN high recognition (ROC-AUC 96.51) [ 24 ] 2021 Image fusion (clinical & dermoscopic) : multi-labeled deep feature extractor and clinically constrained classifier chain (CC) Skin Cancer (Melanoma) publicly available 7-point checklist dataset DCNN, CC, PCA Reported 81.3% accuracy [ 6 ] 2022 Multiclass skin lesion classification using feature fusion & extreme learning machine (ELM) Skin Disease (Skin Lesion) HAM10000 and ISIC2018 SVM, fine KNN, DT, NB, ensemble tree (EBT), & single hidden layer ELM Registered best accuracy of 94.36 percent [ 32 ] 2022 Apply features fusion on manual and automatic feature extraction Skin Cancer DermIS dataset CNN, LSTM, LBP, LBP, Inception V3 Achieved maximum accuracy of 99.4% [ 33 ] 2023 Dual-branch (feature) fusion network using DCNN and Transformer branches for local and global feature extraction Skin Disease (Skin Lesion) Used a private dataset XJUSL DCNN Reducing parameters by 11.17 M improved classification accuracy by 1.08% [ 34 ] 2023 Feature fusion : fast-bounding box (FBB), Hybrid Feature Extractor (HFE), and the CNN VGG19 based CNN Skin Cancer (Melanoma) ISIC 2017, Academic torrents dataset CNN Registered 99.85% accuracy After conducting the final screening procedures, 9 articles have been selected for the final analysis of this systematic review as presented in Table 2 above. The 9 articles selected utilized DL-based methods based on MMDF techniques for the diagnoses of different skin diseases other than NTDs. The 9 studies are selected for the final analysis of this review since there are no similar studies found for the diagnosis of skin related NTDs based on MMDF. Since skin related NTDs are being diagnosed using skin photos or images, patient records and related information, these studies are selected and reviewed to analyze the different techniques utilized by those studies. The final analysis is conducted on the 9 articles using 5 different analysis criteria (the methods used, diseases selected for diagnosis, dataset used, algorithms used and corresponding performance achievements) to identify research gaps as summarized in Table 2 below. Table 2 Summary of the review of the DL-based multimodal data fusion techniques for the diagnosis of skin diseases Ref. Study Method / Approach Used Algorithm(s) Used Performance / Accuracy Results Achieved [ 7 ] Combining images and metadata features CNN: using 5 pre-rained models Performs better than the other combination approaches in 6 out of 10 scenarios. [ 8 ] a naive combination of patient data and an image classifier CNN CNN: AUROC of 92.30% ±0.23% & balanced accuracy of 83.17% ±0.38%), naive strategy: accuracy to 86.72% ±0.36%. [ 9 ] A DNN-based multi-modal classifier using wound images and their locations (AlexNet + MLP, AlexNet + LSTM, ResNet50 + MLP, VGG16 + LSTM) Max. Acc. on mixed class: varies from 82.48 to 100% the max. acc. on wound-class varies from 72.95 to 97.12% in various experiments [ 35 ] 2 imaging modalities with patient metadata CNN, RF classifier, ResNet-50, ILSVRC binary melanoma detection (AUC 0.866 vs 0.784) & multiclass classification (mAP 0.729 vs 0.598) [ 36 ] Multiplication-based DF, using the metadata CNN, the color constancy algorithm outperforms traditional baseline approaches (p-values are smaller than 0.05) [ 37 ] a DNN with two encoders and application of a multimodal fusion module CNN: CNN models (ResNet-50) ACC (0.768 ± 0.022), BACC (0.775 ± 0.022) & outperform other metadata fusion methods (MetaNet (P = 0.035) and MetaBlock (P = 0.028)) [ 38 ] Multimodal Transformer using Vision Transformer (ViT) model CNN: ResNet101, Densenet121) and ViT models Private DS (accuracy: 0.816, which is better than other popular networks) & On ISIC 2018 DS (accuracy: 0.9381 and an AUC of 0.99) [ 39 ] Preprocessing, feature extraction, and classification/diagnosis CNN: 6 CNN pre-trained models with tuning algorithms Av. acc, sensitivity, specificity, precision, & disc similarity coefficient (DSC) of around 99.94%, 91.48%, 98.82%, 97.01%, and 94.00% [ 40 ] fusion of clinical skin image & patient clinical data, feature extraction & attention mechanisms CNN: (VGGNet19, ResNet50, DenseNet121 & Inception-V3) Achieved accuracy of 80.42% (an improvement of about 9% compared with the model accuracy using only medical images) 4.5. Methods used for building diagnostic models for skin diseases In the final analysis of this systematic review, the nine studies identified proposed and demonstrated the MMDF approach for the diagnosis of different skin diseases using their corresponding datasets. The studies utilized different methods and algorithms that include CNN, random forest, multilayer perceptron (MLP), long-short term memory (LSTM), the color constancy algorithm, and hyperparameter optimization (HPO) algorithms. Accordingly, 88.9% of the studies (8 articles) primarily utilized the CNN algorithm along with CNN architectures, while 11.1% of the studies utilized MLP and LSTM along with CNN architectures including ResNet50, VGG16, and AlexNet. In general, the studies employed different methods to demonstrate the DL-based methods for combining different modalities of patient data using different methods, such as the attention-based mechanism for combining images and metadata features, a multimodal transformer using the Vision Transformer (ViT) model, and mapping heterogeneous data features. In addition, DCNN architectures such as Densenet121, ILSVRC 2015, VGG16, VGGNet19, ResNet50, ResNet101, DenseNet121, Inception-V3, AlexNet with MLP, AlexNet with LSTM, ResNet50 with MLP, and ViT models were utilized for feature extraction and transfer learning purposes. 4.6. Fusion strategies suggested for skin disease diagnosis Generally, data fusion techniques determine some issues, including the method of integrating data, the data being fused or integrated, and the level at which data will be integrated. The studies used for this review demonstrated various fusion approaches, mainly feature fusion, model fusion, image fusion, and MMDF techniques. In this regard, 89% of the selected studies analyzed in this review implemented MMDF approaches for integrating mainly clinical images and textual medical data. Whereas only one study (11%) demonstrated the MMDF approach for combining two imaging modalities (dermatoscopic and macroscopic images) with patient metadata [ 35 ]. As reported by the studies used in this review, various fusion strategies have been experimented with on a particular dataset while developing a diagnostic model for specific skin disease(s). Accordingly, the fusion methods or strategies include integrating multiple imaging modalities (2 image modalities in this case) with textual patient data [ 35 ], using a multiplication-based fusion approach (used to control data imbalance) [ 36 ], using the metadata processing block (MetaBlock) for enhancing features extracted from the images throughout the classification [ 7 ], other study used a naive combination of the patient data classifier module and a whole slide image classifier module [ 8 ]. Furthermore, using a DNN that has two encoders for extracting image features and textual features, a MMDF module with intra-modality self-attention and inter-modality cross-attention capability was experimented with, and it was reported that the model outperformed other fusion models [ 37 ]. On the other hand, a neural network with a multimodal transformer consisting of two encoders for both images and metadata and one decoder to fuse the multimodal information using the ViT model to extract image features, a soft label encoder for the metadata, and a mutual attention block to fuse the different features [ 38 ]. In another study, a fusion system was developed using four procedures consisting of preprocessing the image and metadata, feature extraction using six pre-trained models, feature concatenation (using CNN through convolutional, pooling, and auxiliary layers), and finally classification of skin disease [ 39 ]. Similarly, the feature concatenation method was used to develop a wound classifier multimodal network by concatenating the image classifier and location-based classifier outputs [ 9 ]. Finally, a skin cancer diagnostic model was developed following three procedures, including extracting features (skin images and patient clinical data using CNN architectures), using the attention mechanism (for handling the multimodal features), and finally developing a feature fusion model [ 40 ]. 4.7. Achievements of MMDF techniques in diagnosing skin diseases As stated by the studies reviewed, in developing diagnostic models using MMDF techniques for skin diseases, various DL methods and algorithms were used, including CNN, Random Forest, MLP, and LSTM. The algorithms achieved sufficiently higher performances in their respective studies while being tested on a particular dataset. Consequently, it was confirmed that MMDF techniques outperform traditional baseline diagnostic approaches [ 7 ][ 36 ]. Furthermore, the majority of the studies reviewed reported that the disease classification models achieved accuracy of more than 80% [ 8 ][ 9 ][ 35 ][ 38 ]. A study using a DNN with two encoders that implement a multimodal fusion module with intra-modality self-attention and inter-modality cross-attention reported an accuracy of 76.8% [ 37 ]. Similarly, another study used in this review that used medical image analysis based on feature extraction, feature concatenation, and classification or diagnosis methods reported 99.94% accuracy in the classification or diagnosis of seven selected skin diseases. In general, as the analysis results show, MMDF techniques are significantly improving classification accuracies. Therefore, the utilization of multimodal data fusion techniques based on the deep learning methods, algorithms, and models in different settings (such as an ensemble of two or more of those methods, algorithms, and models) is a potential research area that needs further investigation, especially for the diagnosis of NTDs. 5. Discussion The primary goal of this systematic review is to collect and analyze research studies that are pertinent to the area of DL-based models that use multimodal data fusion techniques for the diagnosis of skin related NTDs. The analysis was conducted based on the guiding parameters initially set, which include the DL methods or approaches utilized for the diagnosis of the skin disease, the data fusion methods used, the type of medical data to be combined, the algorithms used, and the performance of each algorithm in the DL-based MMDF skin disease diagnostic model or system. 5.1. Important Findings and Future Directions After collecting and analyzing 427 different articles, the nine articles used for the final review presented important dimensions in the area that include the fusion methods that can be used in different settings. In this regard, different fusion techniques were identified, such as image fusion [ 24 ], feature fusion [ 31 ][ 32 ][ 33 ], multimodal fusion for the combination of two or more modalities of data (such as two or more different modalities of image data) can be integrated with another modality of data, such as patient metadata [ 35 ]. This clearly demonstrated that data fusion techniques can be used to integrate multiple types or modalities of data to enhance the performance of DL models, especially for the diagnosis of skin diseases. In this regard, as most of the NTDs can be diagnosed using multiple types of data coming from different sources, including skin lesions and patient metadata, multimodal data fusion can be a potential approach to be utilized for the diagnosis of NTDs. A multimodal data fusion problem can also be approached from different perspectives. Some suggest a method for fusing data using deep CNN-based encoders and decoders for the extraction of image and metadata features to be combined, as well as using transformer modules [ 37 ][ 38 ]. Generally, the whole MMDF process can be implemented using four different DCNN modules that perform the data feature pre-processing, feature extraction, feature concatenation or combination, and finally disease classification. The very common task for all MMDF tasks is feature extraction, which will be used to extract features from the different modalities of data. Hence, proper feature extraction models and tools should be used for the feature extraction tasks. In deep learning, CNN-based pre-trained modes such as ResNet, VGG15/50, DenseNet, Inception, and similar DCNN architectures can be used for feature extraction. Regarding algorithm utilization for the implementation of MMDF methods for the diagnosis of skin diseases, deep CNN models are by far the most experimented with and successful algorithms used. CNN, along with the DL-based CNN architectures, are playing important roles in the basic task of MMDF methods, feature extraction. Other algorithms, such as SVM, RF, and other ML algorithms, could also be used along with the CNN models. The implementation of MMDF methods using these algorithms achieved outperformance, as demonstrated by the studies analyzed in this review. Hence, as one of the potential beneficiaries, the diagnosis of NTDs can utilize these algorithms and methods to enhance the quality of diagnostic services. As potential feature work, the majority of the studies analyzed in this review share a common drawback, lack of larger datasets of images and metadata for training the intended models. It is crucial that a sufficient amount of quality and a balanced dataset containing the different modalities of data, including the different modalities of images and metadata, should be used to ensure the disease classification accuracy of the models to be developed. Finally, from the strengths and weaknesses observed from the diverse studies analyzed in this systematic review, it was found that no single method of implementing the MMDF method guarantees a 100% achievement. However, with proper experimentation, analysis, and possible integration of one or more of the MMDF techniques presented in the analyzed studies, there is a high potential for enhancing the diagnostic quality of skin diseases. It is, therefore, worthwhile to experiment with and adapt the MMDF techniques for the diagnosis of NTDs, since the majority of NTDs are currently diagnosed using skin signs and symptoms with related patient metadata. 6. Conclusion In this systematic review, articles were collected from seven major and reputed sources where 427 study papers were organized, classified, screened and selected to analyze the application of DL-based diagnostic models using multimodal data fusion techniques for the diagnoses of skin related NTDs. Although there are studies that demonstrate the utilization of DL methods for the diagnoses of NTDs, no previous studies were found regarding the implementation of MMDF methods for the diagnoses of NTDs. Similar studies using MMDF for the diagnoses of other skin diseases, such as skin cancer, are reviewed to extract information about the implementation of these methods. In doing so, the selected studies are analyzed using parameters such as research approaches used, disease(s) selected for the study, the dataset used, algorithms used, the performance achieved, and future directions suggested by the study. Accordingly, although all the reviewed studies used diverse research methods and datasets based on their problem, DL-based CNN algorithms were found to be by far the most frequently used algorithm by all studies reviewed. In addition, DNN-based network architectures were widely utilized. In general, the implementation of MMDF methods for the diagnosis of skin diseases significantly enhances the diagnostic performances of models as per different studies reviewed, as confirmed in this review. Hence, utilizing MMDF methods for the diagnoses of skin diseases, particularly for skin related NTDs, would be paramount towards developing DL-based diagnostic models for NTDs. Declarations Author Contribution All authors contributed equally to this work. References WHO (2020) Ending the neglect to attain the sustainable development goals: a road map for neglected tropical diseases 2021–2030. Page W-Q (2023) accessed Jan. 15, Neglected tropical diseases. https://www.who.int/news-room/questions-and-answers/item/neglected-tropical-diseases World Health Organization, Ending the neglect to attain the Sustainable Development Goals: A rationale for continued investment in tackling neglected tropical diseases 2021–2030 (2022) [Online]. Available: https://apps.who.int/iris/handle/10665/70809 Souza AA et al (2021) Diagnostics and the neglected tropical diseases roadmap: Setting the agenda for 2030. Trans R Soc Trop Med Hyg 115(2):129–135. 10.1093/trstmh/traa118 Abdela SG et al (2020) Looking for NTDs in the skin; an entry door for offering patient centered holistic care. J Infect Dev Ctries 14 6.1, pp. 16S-21S. 10.3855/jidc.11707 Afza F, Sharif M, Khan MA, Tariq U, Yong HS, Cha J (2022) Multiclass Skin Lesion Classification Using Hybrid Deep Features Selection and Extreme Learning Machine. Sensors 22(3). 10.3390/s22030799 Pacheco AGC, Krohling RA (2021) An Attention-Based Mechanism to Combine Images and Metadata in Deep Learning Models Applied to Skin Cancer Classification. IEEE J Biomed Heal Informatics 25(9):3554–3563. 10.1109/JBHI.2021.3062002 Höhn J et al (2021) Combining CNN-based histologic whole slide image analysis and patient data to improve skin cancer classification. Eur J Cancer 149:94–101. 10.1016/j.ejca.2021.02.032 Anisuzzaman DM, Patel Y, Rostami B, Niezgoda J, Gopalakrishnan S, Yu Z (2022) Multi-modal wound classification using wound image and location by deep neural network. Sci Rep 12(1):1–20. 10.1038/s41598-022-21813-0 feng Shen W, wei Tang H, Li Jbo, Li X, Chen S (2023) Multimodal data fusion for supervised learning-based identification of USP7 inhibitors: a systematic comparison. J Cheminform 15(1):1–16. 10.1186/s13321-022-00675-8 Gao J, Li P, Chen Z, Zhang J (2020) A survey on deep learning for multimodal data fusion. Neural Comput 32(5):829–864. 10.1162/neco_a_01273 Lipkova J et al (2022) Artificial intelligence for multimodal data integration in oncology. Cancer Cell 40(10):1095–1110. 10.1016/j.ccell.2022.09.012 [13] N, Luo X, Zhong L, Su Z, Cheng W, Ma, Hao P (2023) Artificial intelligence-assisted dermatology diagnosis: From unimodal to multimodal. Comput Biol Med 165:107413. no. July10.1016/j.compbiomed.2023.107413 Beesetty R et al (2023) Leprosy Skin Lesion Detection: An AI Approach Using Few Shot Learning in a Small Clinical Dataset, Indian J Lepr , vol. pp. 89–102, 2023, [Online]. Available: http://www.ijl.org.in Ali Z et al (2022) A Proposed Framework for Early Prediction of Schistosomiasis. Diagnostics 12(12):1–25. 10.3390/diagnostics12123138 Steyve N, Steve P, Ghislain M, Ndjakomo S, pierre E (September, 2022) Optimized real-time diagnosis of neglected tropical diseases by automatic recognition of skin lesions. Inf Med Unlocked 33. 10.1016/j.imu.2022.101078 Yotsu RR, Ding Z, Hamm J, Blanton RE (2023) Deep learning for AI-based diagnosis of skin-related neglected tropical diseases: A pilot study. PLoS Negl Trop Dis 17(8):1–12. 10.1371/journal.pntd.0011230 Castanedo F (2013) A review of data fusion techniques, Sci. World J. , vol. 2013, 10.1155/2013/704504 Shoumy NJ, Ang LM, Seng KP, Rahaman DMM, Zia T (2020) Multimodal big data affective analytics: A comprehensive survey using text, audio, visual and physiological signals. J Netw Comput Appl 149:102447. 10.1016/j.jnca.2019.102447 Zhang T et al (2021) A Feature Fusion Method with Guided Training for Classification Tasks, Comput. Intell. Neurosci. , vol. no. c, 2021, 10.1155/2021/6647220 Lu X, Duan X, Mao X, Li Y, Zhang X (2017) Feature Extraction and Fusion Using Deep Convolutional Neural Networks for Face Detection, Math. Probl. Eng. , vol. 2017, 10.1155/2017/1376726 AlDahoul N, Karim HA, Ba Wazir AS (2021) Model fusion of deep neural networks for anomaly detection. J Big Data 8(1). 10.1186/s40537-021-00496-w Kaur H, Koundal D, Kadyan V (2021) Image Fusion Techniques: A Survey. Arch Comput Methods Eng 28(7):4425–4447. 10.1007/s11831-021-09540-7 Wang Y, Cai J, Louie DC, Wang ZJ, Lee TK, June (2021) 104812 doi: 10.1016/j.compbiomed.2021.104812 An I, Harman M, Ibiloglu I (2017) Topical Ciclopirox Olamine 1%: Revisiting a Unique Antifungal. Indian Dermatol Online J 10(4):481–485. 10.4103/idoj.IDOJ Yélamos O, Mary Diem L, Braun RP, French KK, Marghoob AA (2019) Dermoscopy for Dermatopathologists. Elsevier Inc.. 10.1016/B978-0-323-37457-6.00028-6 Pawłowski M, Wróblewska A, Sysko-Romańczuk S (2023) Sensors 23(5):1–16. 10.3390/s23052381 . Effective Techniques for Multimodal Data Fusion: A Comparative Analysis Page MJ et al (2021) The prisma 2020 statement: An updated guideline for reporting systematic reviews. Med Flum 57(4):444–465. 10.21860/medflum2021_264903 PRISMA, PRISMA (2023) : TRANSPARENT REPORTING of SYSTEMATIC REVIEWS and META-ANALYSES. http://www.prisma-statement.org/ (accessed Nov 15, AI R (2023) accessed Oct. 20, RAYYAN. https://www.rayyan.ai / Bakkouri I, Afdel K (2020) Computer-aided diagnosis (CAD) system based on multi-layer feature fusion network for skin lesion recognition in dermoscopy images. Multimed Tools Appl 79:29–30. 10.1007/s11042-019-07988-1 Mahum R, Aladhadh S (2022) Skin Lesion Detection Using Hand-Crafted and DL-Based Features Fusion and LSTM, Diagnostics , vol. 12, no. 12, 10.3390/diagnostics12122974 Zhang D, Li A, Wu W, Yu L, Kang X, Huo X (2023) CR-Conformer: a fusion network for clinical skin lesion classification. Med Biol Eng Comput 012345678910.1007/s11517-023-02904-0 Afra S, Alhajj R (2019) al Pr p ro of. Phys A 123151. 10.1016/j.jpi.2023.100341 Yap J, Yolland W, Tschandl P (2018) Multimodal skin lesion classification using deep learning. Exp Dermatol 27(11):1261–1267. 10.1111/exd.13777 Li W, Zhuang J, Wang R, Zhang J, FUSING METADATA AND DERMOSCOPY IMAGES FOR SKIN, DISEASE DIAGNOSIS School of Data and Computer Science (2020) Sun Yat-sen University, China Key Laboratory of Machine Intelligence and Advanced Computing, MOE, Guangzhou. China Department of Computer Science an, pp 1996–2000 Ou C et al (2022) A deep learning based multimodal fusion model for skin lesion diagnosis using smartphone collected clinical images and metadata, Front. Surg. , vol. 9, no. October, pp. 1–9, 10.3389/fsurg.2022.1029991 Cai G, Zhu Y, Wu Y, Jiang X, Ye J, Yang D (2023) A multimodal transformer to fuse images and metadata for skin disease classification. Vis Comput 39(7):2781–2793. 10.1007/s00371-022-02492-4 Almuayqil SN, Abd El-Ghany S, Elmogy M (2022) Computer-Aided Diagnosis for Early Signs of Skin Diseases Using Multi Types Feature Fusion Based on a Hybrid Deep Learning Model. Electron 11(23). 10.3390/electronics11234009 Chen Q, Li M, Chen C, Zhou P, Lv X, Chen C (2023) MDFNet: application of multimodal fusion method based on skin image and clinical data to skin cancer classification. J Cancer Res Clin Oncol 149(7):3287–3299. 10.1007/s00432-022-04180-1 Tables Table 3 is available in the Supplementary Files section. Additional Declarations No competing interests reported. Supplementary Files Appendix.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3870993","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":267607745,"identity":"50ce75b1-3b7c-4a2c-9372-041591142268","order_by":0,"name":"Yohannes Minyilu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABGklEQVRIiWNgGAWjYFACHhiDsYGBwcCCgR/CYyZaiwSDZANcCy5tPCg8CQaDAwS0GBzvPfi5cIeNPYPY4bbHPAUSiZuPn06TYKiwTmxg7z+AVcuZc8nSM8+kJTZIJ7Yb8xhIJG47k7tNguFMemIDz2HsttzIMZDmbTucwCCd2CYN1nIAqIWx7XBig0QyLi3Gv3nb/tvDtWzufwvU8g+oRf4xLi1mQFsOMDbAtGyQANnSALIFu/clz5wxs+ZtSwaqT2yTnGMgYTzjxtvNFgnH0o3beJINsGnhO95jfJu3zc6eXzr9mcSbPzay/f25G298qLGW7Wc/+ACbFoUDUAYbEDPB4ygBKoINyDcgcRh/4FA1CkbBKBgFIxsAAGjAXLbgOCiUAAAAAElFTkSuQmCC","orcid":"","institution":"Arba Minch University","correspondingAuthor":true,"prefix":"","firstName":"Yohannes","middleName":"","lastName":"Minyilu","suffix":""},{"id":267607746,"identity":"781076c9-71b8-43b5-adbb-0ebdb3c602aa","order_by":1,"name":"Mohammed Abebe","email":"","orcid":"","institution":"Arba Minch University","correspondingAuthor":false,"prefix":"","firstName":"Mohammed","middleName":"","lastName":"Abebe","suffix":""},{"id":267607747,"identity":"7a26c237-621f-4a6b-b759-828ba1c8b38d","order_by":2,"name":"Million Meshesha","email":"","orcid":"","institution":"Addis Ababa University","correspondingAuthor":false,"prefix":"","firstName":"Million","middleName":"","lastName":"Meshesha","suffix":""}],"badges":[],"createdAt":"2024-01-16 21:29:18","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3870993/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3870993/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":49882647,"identity":"180956c2-101a-4fe5-a894-fb10b6757750","added_by":"auto","created_at":"2024-01-19 16:39:52","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":45214,"visible":true,"origin":"","legend":"\u003cp\u003eArticle selection for systematic literature review following PRISMA flow chart\u003c/p\u003e","description":"","filename":"floatimage246.png","url":"https://assets-eu.researchsquare.com/files/rs-3870993/v1/85dd8dcce9e872eca7d939da.png"},{"id":49882649,"identity":"5faf8f24-789d-40e7-b63a-a24e71988fe0","added_by":"auto","created_at":"2024-01-19 16:39:53","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":61788,"visible":true,"origin":"","legend":"\u003cp\u003eTotal collected articles and their distribution by article databases/ search engines\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-3870993/v1/a5835f5993d52226546fe2a6.png"},{"id":49883354,"identity":"dce9a222-5144-4a79-9b3b-dbb7802c0b9b","added_by":"auto","created_at":"2024-01-19 16:47:53","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":100041,"visible":true,"origin":"","legend":"\u003cp\u003eDistribution of articles screened by databases\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-3870993/v1/3a67afb8d49e6f17a7814124.png"},{"id":49882652,"identity":"1ebfb6dc-fd31-431f-b198-8d3169dc2917","added_by":"auto","created_at":"2024-01-19 16:39:53","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":65160,"visible":true,"origin":"","legend":"\u003cp\u003eDistribution of articles after the third level screening by publication year\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-3870993/v1/202c1363eeba680e59d6d364.png"},{"id":49882650,"identity":"f735a67a-6e0b-4b44-91de-c1f62668543e","added_by":"auto","created_at":"2024-01-19 16:39:53","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":74604,"visible":true,"origin":"","legend":"\u003cp\u003eDistribution of articles by research area/methods and selected diseases after the third level screening\u003c/p\u003e","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-3870993/v1/ce914a70542df012370a7c24.png"},{"id":50630867,"identity":"94fa7238-5f45-453c-9371-9fbaa17dc14a","added_by":"auto","created_at":"2024-02-04 23:39:57","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":983750,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3870993/v1/33e0c27f-8410-47a9-a552-9fa1ae5edecf.pdf"},{"id":49882648,"identity":"860eff6d-3321-4f03-86f8-2ab6cfcacb23","added_by":"auto","created_at":"2024-01-19 16:39:52","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":39941,"visible":true,"origin":"","legend":"","description":"","filename":"Appendix.docx","url":"https://assets-eu.researchsquare.com/files/rs-3870993/v1/eac1f464b187dc60075e6545.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Applying Multimodal Data Fusion based on Deep Learning Methods for the Diagnosis of Neglected Tropical Diseases: A Systematic Review","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eBeing the largest organ in the human body, the skin can serve as an indicator of some illnesses arising from different causes such as cancer, internal organ failure, and neglected tropical diseases. NTDs are the most prevalent diseases globally affecting more than one billion people worldwide (i.e., more than ten percent of the world\u0026rsquo;s population), particularly, in the tropical areas of the world among the poorest, most vulnerable and outcast groups and still have devastating impacts on people's physical, mental, and social well-being [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e][\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e][\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e][\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. However, these diseases can be diagnosed using skin related symptoms since majority of the NTDs have primary skin indicators or associated clinical features where 18 of the 20 NTDs (recognized by the World Health Organization (WHO)) having skin related symptoms [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. Hence, the utilization of DL-based diagnostic systems for the diagnoses and recognition of skin related NTDs will be a great achievement in overcoming the NTDs. This study endeavored to present a thorough systematic review of studies regarding the implementation of MMDF techniques for the diagnoses of skin related NTDs. Since no previous studies implemented MMDF techniques for the diagnoses of skin NTDs, related studies conducted for the diagnoses of skin diseases other than NTDs using MMDF and DL methods were deeply appraised by this review. These studies confirmed that the utilization of MMDF techniques outperforms the traditional diagnostic models that implemented DL methods without MMDF [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e][\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e][\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e][\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. It is in view of these facts the study is -motivated to conduct this systematic literature review and a thorough appraisal of previous studies using the PRISMA method of systematic review based on the following guiding questions:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eWhat DL methods or approaches were utilized for the diagnoses of the skin disease(s)?\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eWhich data fusion methods were used for the skin disease diagnosis tasks?\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eWhat types of medical data were integrated to demonstrate MMDF method for the diagnoses of the skin diseases?\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eWhich algorithms were used and how does each algorithm perform in the DL-based MMDF skin disease diagnostic model or system?\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e"},{"header":"2. The Need for Intelligent Diagnostic Systems","content":"\u003cp\u003eIn recent times, due to the high desire to enhance the diagnostic processes in the healthcare sectors, the utilization of automated and intelligent diagnostic systems is getting greater attention for the diagnosis of various diseases. In this regard, intelligent diagnostic systems built based on machine learning (ML) and deep learning (DL) methods are the most researched and deployed approaches in the healthcare sector to support diagnostic decision making. On the other hand, in the real-world clinical settings, efficient disease diagnostic processes are basically carried out by using different clinical data that are taken from different sources and different formats or modalities including textual patient information and medical clinical images such as X-ray, dermoscopic images or even patient skin images. The integrative utilization of the diverse modalities of medical data can be used to enhance the diagnostic processes, thereby enhancing the quality of healthcare services, by using the ML and DL methods. In ML, this process of integrating multiple modalities of data (possibly taken from different sources) is technically called multimodal data fusion [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e][\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Multimodal data techniques are playing vital roles in developing intelligent disease diagnostic systems for different diseases such as in dermatology [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. In this regard, MMDF techniques are advancing diagnostic accuracy, where these methods outperform other baseline methods, as presented in [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e].\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1. Deep Learning and Diagnoses of NTDs\u003c/h2\u003e \u003cp\u003eThe current diagnostic approaches used for NTDs are mainly based on clinical procedures, such as patient observation and laboratory examinations based on limited resources in most affected areas. Currently, however, there are efforts towards the utilization of intelligent diagnostic tools using ML and DL approaches. Since most of the NTDs are curtly being diagnosed using skin manifestation, the utilization of DL-based approaches for the diagnoses of these diseases would be a great potential to support and enhance the diagnostic processes. In this regard, different studies were previously conducted to diagnose various NTDs. (Beesetty et al. 2023) [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e], conducted a study towards leprosy skin lesion detection employing a Siamese (Siamese NN)-based few-shot learning (FSL) model for a small clinical dataset and claimed a higher diagnostic accuracy. On the other hand, (Ali et al. 2022) [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e], used ML methods for early prediction of Schistosomiasis and concluded with the CatBoost model showing the best performance with the highest accuracy being above 80%. An optimized diagnostic approach was also proposed for NTDs by selecting three diseases and developing a model using SVM and the black hole algorithm (BHO) achieving more than 90% accuracy [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. Another study reviewed by the current study demonstrated a DL-based diagnostic model for NTDs using skin images only and achieved 70% accuracy [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. All the aforementioned studies utilized ML and DL methods for the diagnoses of NTDs and achieved remarkable results in terms of accuracy. However, no previous studies were found that utilized DL-based methods using MMDF techniques which will help to achieve higher diagnostic accuracies, as experimented in other studies for non-NTD skin diseases which require further research.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2. Data Fusion Approaches\u003c/h2\u003e \u003cp\u003eData or information fusion represents the usage of data or information from different sources in different formats or modalities for interpretation in all tasks that require any type of parameter estimation or prediction using data or information [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. There are different fusion techniques to combine and aggregate multimodal data which include feature-level fusion, decision-level or late fusion, hybrid multimodal fusion, model-level fusion, rule-based fusion, classification-based fusion and estimation-based fusion [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e].\u003c/p\u003e \u003cdiv id=\"Sec5\" class=\"Section3\"\u003e \u003ch2\u003e2.2.1. Feature Fusion\u003c/h2\u003e \u003cp\u003eFeature fusion is a data integration technique used to aggregate multiple feature sets extracted from multiple input data to generate a single feature set [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. In image processing problems, it refers to the fusion of feature vectors of training images extracted from shared weight network layer and feature vectors composed of other numerical data [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. It helps to learn image features fully for the description of their rich internal information [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. Various studies are found and appraised that use feature fusion techniques to develop diagnostic models for the diagnoses of skin diseases.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section3\"\u003e \u003ch2\u003e2.2.2. Model Fusion\u003c/h2\u003e \u003cp\u003eModel fusion, also known as late fusion, represents a fusion approach that combines different models. The study done by (AlDahoul et al 2021), [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e], combines two deep neural networks including binary normal/attack classifier and multi-attack classifier to train a deep neural network (DNN) for network anomaly detection. As mentioned by (Shoumy et al 2020) [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e], the model fusion technique uses the connection between experimental data under different modalities.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section3\"\u003e \u003ch2\u003e2.2.3. Image Fusion\u003c/h2\u003e \u003cp\u003eImage fusion combines different images and generates informative images by integrating images obtained from different sources [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. A previous study by (Y. Wang et al 2021), [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e], suggested that aggregating medical images helps to enhance diagnostic accuracy. This claim was demonstrated by fusing clinical images and dermoscopic images using the deep convolutional neural networks (DCNN) methods and achieved an overall accuracy above 80%. While the clinical images are clinically captured photographs [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e], dermoscopic images represent images taken by dermatologists using dermoscopy [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e].\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section3\"\u003e \u003ch2\u003e2.2.4. Multimodal Data Fusion\u003c/h2\u003e \u003cp\u003eMultimodal data represents the different formats or modalities of data such as text, image, video, and audio. A multimodal data fusion approach is used for combining particular modalities to derive multimodal representation [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e][\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e][\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e][\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. This approach has multiple applications for healthcare systems as it allows the combination of different modalities of data, such as textual medical history of patients, clinical images of patients (such as skin images of patients) to form a single multimodal dataset that can be used to train diagnostic models using ML and DL methods. In this regard, various studies implemented and demonstrated MMDF for the diagnoses of different skin diseases as summarized in Table\u0026nbsp;3, \u003cspan refid=\"Sec25\" class=\"InternalRef\"\u003eAppendix\u003c/span\u003e.\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"3. Materials and Methods","content":"\u003cp\u003eTo conduct and report this systematic review, we follow as a basis, the steps suggested by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) model [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e][\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e] as described below.\u003c/p\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e3.1. Search Strategy\u003c/h2\u003e \u003cp\u003eResearch articles were searched from all the major indexing databases and search engines, such as Google Scholar, PubMed, IEEE Explore, and ScienceDirect. In addition, proper search keywords were prepared and used while searching for the articles, where the search keywords have the appropriate level of relationships with the topics and contents of the articles. A set of searching keywords have been used to deeply search and filter the articles. In this regard, Boolean operators \u0026ldquo;AND\u0026rdquo; and \u0026ldquo;OR\u0026rdquo; were mainly used. The \u0026ldquo;AND\u0026rdquo; operator was used to search for articles in a specific research area to narrow down the search results; \u0026ldquo;multimodal medical data\u0026rdquo; AND \u0026ldquo;data fusion\u0026rdquo; to search for articles containing both phrases. On the other hand, the \u0026ldquo;OR\u0026rdquo; Boolean operator was used to search for articles from wider perspectives as this operator broadens the search results such as \u0026ldquo;machine learning\u0026rdquo; OR \u0026ldquo;deep learning\u0026rdquo;.\u003c/p\u003e \u003cp\u003eUsing the specified methods and operators, multiple search keywords were initially prepared and used to find a sufficient amount of relevant articles. Although a lot of search keywords were used while searching for the articles, some of the keywords include [\"Neglected Tropical Diseases\" OR \"NTDs\" AND \"Diagnosis\" OR \"diagnostic model\" AND \"Deep Learning\" OR \"DL\" OR \"Convolutional Neural Network\" OR \"CNN\" OR \"Deep Neural Network\" OR \"DNN\" OR \"Recurrent Neural Network\" OR \"RNN\"], [\"Neglected Tropical Diseases\" OR \"NTDs\" AND \"Diagnosis\" OR \"diagnostic model\" AND \"Deep Learning\" OR \"DL\" or \"Convolutional Neural Network\" OR \"CNN\" OR \"Deep Neural Network\" OR \"DNN\" OR \"Recurrent Neural Network\" OR \"RNN\" AND \"Data Fusion\" OR \"Multimodal medical Data\" OR \"Multimodal Data Fusion\"], [(((deep learning) AND ((diagnostic model) OR (diagnostic system) OR (diagnostic tool)) AND (skin diseases) AND (skin images)) AND (medical record)) AND (data fusion))].\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e3.2. Eligibility Criteria\u003c/h2\u003e \u003cp\u003eNot all articles are critically relevant for the review concerning the integration of multimodal data fusion techniques based on DL methods for the diagnosis of skin related NTDs. Hence, a set of inclusion and/or exclusion criteria are applied, as shown below.\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eArticles must demonstrate DL methods for the diagnosis of skin NTDs, at least skin diseases if not implemented for skin NTDs, with proper evaluation of the methods used.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eArticles must demonstrate proper utilization of multimodal data fusion techniques for the diagnosis of skin NTDs, at least skin diseases since there are no previous studies that use multimodal data fusion techniques for the diagnosis of skin NTDs so far.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eArticles must incorporate precise presentation or discussion and evaluation of all the methods and techniques used in that particular article.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eAn Article that used DL methods for the diagnosis of skin related diseases other than NTDs is selected if that particular article uses new or emerging DL methods and presents a proper analysis of the methods and techniques used for the diagnosis of that particular skin disease(s). However, articles that use the popular and previously used DL methods for the diagnosis of diseases other than skin diseases have fewer chances to be selected.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eThe article should have an appropriate level of similarity and relationships in its topics and contents with the searching keywords used to deeply search and filter the articles.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eArticles that do not utilize DL and data fusion techniques are excluded from analysis.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eArticles published in languages other than English are excluded from analysis.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eFinally, articles published prior to the year 2014 are also excluded.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e3.3. Article Search\u003c/h2\u003e \u003cp\u003eDifferent searching methods such as \u0026lsquo;basic search\u0026rsquo; and \u0026lsquo;advanced search\u0026rsquo; methods were used on multiple article sources. First, the ordinary or basic searching method was used where general titles and the proposed keywords were entered in the regular \u0026lsquo;search box\u0026rsquo; of each of the databases and searched. Secondly, the \u0026lsquo;advanced search\u0026rsquo; option was used which allows to specify subject areas, related topics, publication dates and other relevant options which helps to obtain articles that are relevant to the topic by narrowing down the search results. Using both of the search methods and search keywords, a thorough and rigorous searching was conducted on multiple search engines, journals, databases and libraries to find relevant articles. The sources include Google Scholar, IEEE Explore, MDPI, Mendeley, Nature, PubMed, ScienceDirect, AJOL, IDP, NCBI, PLOS, Springer, and Tropical Medicine and Health. Finally, by specifying article publication dates and applying the searching methods on the different databases, 427 articles that were published between the year 2014 and 2024 were collected and prepared for screening. Each database was used independently to search articles. Furthermore, previously searched sources such as academic web portals, academic libraries and research sites were also used as there were relevant documents from these sources. Hence, 16 articles were collected from such sources. However, no relevant articles were found related to this particular area.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e3.4. Relevant Article Selection\u003c/h2\u003e \u003cp\u003eTo select relevant articles, an extensive searching method was used using wider options of searching keywords. The entire process of article selection for this review was conducted based on the PRISMA method as it was an evidence-based minimum set of items for reporting for systematic reviews and meta-analyses [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e][\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. This was done to provide insights for the recent and future research regarding the utilization of DL methods for NTDs diagnosis, the integration of data fusion techniques if they have been used for NTDs and finally to assess and present feedback so as to enhance the performances of such DL-based models.\u003c/p\u003e \u003cp\u003eTo select relevant articles for this review, five levels of screening were performed where the first level screening was conducted manually using file names and titles. Then, the next levels of screenings were performed using software tools such as \u0026lsquo;EndNote\u0026rsquo; and \u0026lsquo;Rayyan\u0026rsquo;. As a reference management tool, EndNote was used to create a library containing the collected articles and for manipulation and data processing to check duplicate files in the library. Finally, regarding the screening process, using a higher-level screening software tool is mandatory, and for this purpose, Rayyan, a free online software tool [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e] was employed which is mainly used to speed up the literature screening process in systematic reviews. This online tool uses the article library exported from EndNote and it was first used to check duplication based on title, author names and abstracts.\u003c/p\u003e \u003c/div\u003e"},{"header":"4. Results and Analysis","content":"\u003cp\u003eA series of screening operations were implemented on the collected articles in order to identify the most relevant set of articles for this review. In this case, the first level screening was conducted manually on a total of 427 files using file names and titles of the article which allowed as collecting 397 items were selected out of the total 427 articles. Using EndNote to create a library resulted in the automatic removal of 25 articles as there were duplicate files from different folders, followed by an automatic duplicate detection, leading to a library containing 371 articles. Further screening using \u0026lsquo;Rayyan\u0026rsquo;, 4 duplicated articles were detected in the library and two of them were removed where 369 articles were finally identified. Further, using this online software tool, 90 articles that have a relationship with the current topic of the study were selected based on title and abstract analysis. Additional screening was required to identify articles in relation to the study area and 18 articles were identified out of the 90 related articles. Finally, 9 articles were selected for the final analysis. The overall article selection procedure is outlined using the PRISMA flow chart as depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e below.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e4.1. Distribution of Articles\u003c/h2\u003e \u003cp\u003eBy applying the specified searching methods on the seven different databases, 427 articles that were published between the year 2014 and 2024 were collected as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e below. Accordingly, Google Scholar was primarily used and it allowed us to collect 178 articles from the different sources including IEEE Explore, MDPI, Mendeley, Nature, PubMed, ScienceDirect, AJOL, IDP, NCBI, PLOS, Springer, and Tropical Medicine and Health.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eFurthermore, additional analysis was performed regarding the sources of the articles with respect to the first two consecutive initial levels of screening, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e below.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003e4.2. Distribution of Articles by Publication Year\u003c/h2\u003e \u003cp\u003eAfter conducting three levels of screening, 90 articles that have direct relationship with the current systematic review have been selected for further screening based on full-text reading and analyses. The selected articles and their respective publication year along with the distribution of the publications years have been shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e below.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eAs shown, the articles used for this systematic review included studies that have been published recently, where the majority of the studies representing 31% are articles published in 2023, 25% were published in 2022, 16% were published in 2021, 14% were published in 2020, and the remaining 14% were articles published from 2014\u0026ndash;2019.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003e4.3. Distribution of Articles by Methods Used\u003c/h2\u003e \u003cp\u003eFinally, the 90 articles were further analyzed by categorizing them into four different groups, (i) articles that utilized DL methods for the diagnosis of skin diseases, (ii) articles that implement ML \u0026amp; DL techniques for the diagnosis of NTDs, (iii) articles about the implementation of multimodal data fusion techniques for medical data fusion, and (iv), articles that implement multimodal data fusion based on DL-based methods for the diagnosis of skin diseases as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e below. As portrayed in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e below, 54.44% of articles utilized ML and DL methods for the diagnosis of skin diseases in general, 20% deal with multimodal data fusion techniques for healthcare systems and 20% implementation of DL-based multimodal data fusion methods for the diagnosis of skin diseases. On the other hand, 5.56% of the articles utilized ML and DL methods for the diagnoses of NTDs in general have been identified and analyzed. However, no article has been found that deals with the implementation of DL-based MMDF methods for the diagnosis of NTDs which has led to the analyses of previous studies that used this approach for the diagnoses of different skin diseases other than the NTDs. By conducting the fourth level screening, 18 articles that utilize different fusion techniques for the diagnosis of various skin diseases have been identified.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003e4.4. Analysis of Fusion Techniques Used\u003c/h2\u003e \u003cp\u003eThe final screening has resulted in the separation of 7 of the 18 articles due to the fusion techniques they utilize for the diagnosis of skin diseases. The fusion techniques presented in those 7 studies are feature fusion (5 studies), image fusion (1 study) and model fusion (1 review study) as presented in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e below. Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e presented the analysis of three different types of fusion other than MMDF using five different parameters as shown in the table below.\u003c/p\u003e \u003cp\u003eOn the other hand, 2 articles presented a review of the multimodal data fusion techniques for the diagnoses of skin diseases other than NTDs. Although the 2 articles [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e][\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e], didn\u0026rsquo;t implement MMDF techniques for a specific skin disease diagnosis using their datasets of preferences, they presented theoretical analyses. All in all, 9 articles are used for the final analysis of this review.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eReview of the future fusion and related techniques for skin disease diagnoses\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRef\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePub.\u003c/p\u003e \u003cp\u003eYr.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eStudy Method / Approach Used\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eDisease(s) Selected\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eDataset(s) Used\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAlgorithm(s) Used\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003ePerformance Results Achieved\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2019\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTransfer Learning and multi-layer \u003cb\u003efeature fusion\u003c/b\u003e network\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSkin Lesion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eHAM10000 dataset\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eCNN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ehigh recognition (ROC-AUC 96.51)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2021\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003eImage fusion (clinical \u0026amp; dermoscopic)\u003c/b\u003e: multi-labeled deep feature extractor and clinically constrained classifier chain (CC)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSkin Cancer (Melanoma)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003epublicly available 7-point checklist dataset\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eDCNN, CC, PCA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eReported 81.3% accuracy\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2022\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMulticlass skin lesion classification using \u003cb\u003efeature fusion\u003c/b\u003e \u0026amp; extreme learning machine (ELM)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSkin Disease (Skin Lesion)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eHAM10000 and ISIC2018\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eSVM, fine KNN, DT, NB, ensemble tree (EBT), \u0026amp; single hidden layer ELM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eRegistered best accuracy of 94.36 percent\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2022\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003eApply features fusion on\u003c/b\u003e manual and automatic feature extraction\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSkin Cancer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eDermIS dataset\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eCNN, LSTM, LBP, LBP, Inception V3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eAchieved maximum accuracy of 99.4%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2023\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003eDual-branch (feature) fusion\u003c/b\u003e network using DCNN and Transformer branches for local and global feature extraction\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSkin Disease (Skin Lesion)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eUsed a private dataset XJUSL\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eDCNN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eReducing parameters by 11.17 M improved classification accuracy by 1.08%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2023\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003eFeature fusion\u003c/b\u003e: fast-bounding box (FBB), Hybrid Feature Extractor (HFE), and the CNN VGG19 based CNN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSkin Cancer (Melanoma)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eISIC 2017, Academic torrents dataset\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eCNN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eRegistered 99.85% accuracy\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eAfter conducting the final screening procedures, 9 articles have been selected for the final analysis of this systematic review as presented in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e above. The 9 articles selected utilized DL-based methods based on MMDF techniques for the diagnoses of different skin diseases other than NTDs. The 9 studies are selected for the final analysis of this review since there are no similar studies found for the diagnosis of skin related NTDs based on MMDF. Since skin related NTDs are being diagnosed using skin photos or images, patient records and related information, these studies are selected and reviewed to analyze the different techniques utilized by those studies. The final analysis is conducted on the 9 articles using 5 different analysis criteria (the methods used, diseases selected for diagnosis, dataset used, algorithms used and corresponding performance achievements) to identify research gaps as summarized in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e below.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSummary of the review of the DL-based multimodal data fusion techniques for the diagnosis of skin diseases\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRef.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStudy Method / Approach Used\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAlgorithm(s) Used\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePerformance / Accuracy Results Achieved\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCombining images and metadata features\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN: using 5 pre-rained models\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePerforms better than the other combination approaches in 6 out of 10 scenarios.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ea naive combination of patient data and an image classifier\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eCNN: AUROC of 92.30% \u0026plusmn;0.23% \u0026amp; balanced accuracy of 83.17% \u0026plusmn;0.38%), naive strategy: accuracy to 86.72% \u0026plusmn;0.36%.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eA DNN-based multi-modal classifier using wound images and their locations\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(AlexNet\u0026thinsp;+\u0026thinsp;MLP, AlexNet\u0026thinsp;+\u0026thinsp;LSTM, ResNet50\u0026thinsp;+\u0026thinsp;MLP, VGG16\u0026thinsp;+\u0026thinsp;LSTM)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMax. Acc. on mixed class: varies from 82.48 to 100% the max. acc. on wound-class varies from 72.95 to 97.12% in various experiments\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2 imaging modalities with patient metadata\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN, RF classifier, ResNet-50, ILSVRC\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ebinary melanoma detection (AUC 0.866 vs 0.784) \u0026amp; multiclass classification (mAP 0.729 vs 0.598)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMultiplication-based DF, using the metadata\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN, the color constancy algorithm\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eoutperforms traditional baseline approaches (p-values are smaller than 0.05)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ea DNN with two encoders and application of a multimodal fusion module\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN: CNN models (ResNet-50)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eACC (0.768\u0026thinsp;\u0026plusmn;\u0026thinsp;0.022), BACC (0.775\u0026thinsp;\u0026plusmn;\u0026thinsp;0.022) \u0026amp; outperform other metadata fusion methods (MetaNet (P\u0026thinsp;=\u0026thinsp;0.035) and MetaBlock (P\u0026thinsp;=\u0026thinsp;0.028))\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMultimodal Transformer using Vision Transformer (ViT) model\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN: ResNet101, Densenet121) and ViT models\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePrivate DS (accuracy: 0.816, which is better than other popular networks) \u0026amp; On ISIC 2018 DS (accuracy: 0.9381 and an AUC of 0.99)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePreprocessing, feature extraction, and classification/diagnosis\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN: 6 CNN pre-trained models with tuning algorithms\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAv. acc, sensitivity, specificity, precision, \u0026amp; disc similarity coefficient (DSC) of around 99.94%, 91.48%, 98.82%, 97.01%, and 94.00%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003efusion of clinical skin image \u0026amp; patient clinical data, feature extraction \u0026amp; attention mechanisms\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN: (VGGNet19, ResNet50, DenseNet121 \u0026amp; Inception-V3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAchieved accuracy of 80.42% (an improvement of about 9% compared with the model accuracy using only medical images)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec19\" class=\"Section2\"\u003e \u003ch2\u003e4.5. Methods used for building diagnostic models for skin diseases\u003c/h2\u003e \u003cp\u003eIn the final analysis of this systematic review, the nine studies identified proposed and demonstrated the MMDF approach for the diagnosis of different skin diseases using their corresponding datasets. The studies utilized different methods and algorithms that include CNN, random forest, multilayer perceptron (MLP), long-short term memory (LSTM), the color constancy algorithm, and hyperparameter optimization (HPO) algorithms. Accordingly, 88.9% of the studies (8 articles) primarily utilized the CNN algorithm along with CNN architectures, while 11.1% of the studies utilized MLP and LSTM along with CNN architectures including ResNet50, VGG16, and AlexNet. In general, the studies employed different methods to demonstrate the DL-based methods for combining different modalities of patient data using different methods, such as the attention-based mechanism for combining images and metadata features, a multimodal transformer using the Vision Transformer (ViT) model, and mapping heterogeneous data features. In addition, DCNN architectures such as Densenet121, ILSVRC 2015, VGG16, VGGNet19, ResNet50, ResNet101, DenseNet121, Inception-V3, AlexNet with MLP, AlexNet with LSTM, ResNet50 with MLP, and ViT models were utilized for feature extraction and transfer learning purposes.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec20\" class=\"Section2\"\u003e \u003ch2\u003e4.6. Fusion strategies suggested for skin disease diagnosis\u003c/h2\u003e \u003cp\u003eGenerally, data fusion techniques determine some issues, including the method of integrating data, the data being fused or integrated, and the level at which data will be integrated. The studies used for this review demonstrated various fusion approaches, mainly feature fusion, model fusion, image fusion, and MMDF techniques. In this regard, 89% of the selected studies analyzed in this review implemented MMDF approaches for integrating mainly clinical images and textual medical data. Whereas only one study (11%) demonstrated the MMDF approach for combining two imaging modalities (dermatoscopic and macroscopic images) with patient metadata [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eAs reported by the studies used in this review, various fusion strategies have been experimented with on a particular dataset while developing a diagnostic model for specific skin disease(s). Accordingly, the fusion methods or strategies include integrating multiple imaging modalities (2 image modalities in this case) with textual patient data [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e], using a multiplication-based fusion approach (used to control data imbalance) [\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e], using the metadata processing block (MetaBlock) for enhancing features extracted from the images throughout the classification [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], other study used a naive combination of the patient data classifier module and a whole slide image classifier module [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Furthermore, using a DNN that has two encoders for extracting image features and textual features, a MMDF module with intra-modality self-attention and inter-modality cross-attention capability was experimented with, and it was reported that the model outperformed other fusion models [\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e]. On the other hand, a neural network with a multimodal transformer consisting of two encoders for both images and metadata and one decoder to fuse the multimodal information using the ViT model to extract image features, a soft label encoder for the metadata, and a mutual attention block to fuse the different features [\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e]. In another study, a fusion system was developed using four procedures consisting of preprocessing the image and metadata, feature extraction using six pre-trained models, feature concatenation (using CNN through convolutional, pooling, and auxiliary layers), and finally classification of skin disease [\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e]. Similarly, the feature concatenation method was used to develop a wound classifier multimodal network by concatenating the image classifier and location-based classifier outputs [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. Finally, a skin cancer diagnostic model was developed following three procedures, including extracting features (skin images and patient clinical data using CNN architectures), using the attention mechanism (for handling the multimodal features), and finally developing a feature fusion model [\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e].\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec21\" class=\"Section2\"\u003e \u003ch2\u003e4.7. Achievements of MMDF techniques in diagnosing skin diseases\u003c/h2\u003e \u003cp\u003eAs stated by the studies reviewed, in developing diagnostic models using MMDF techniques for skin diseases, various DL methods and algorithms were used, including CNN, Random Forest, MLP, and LSTM. The algorithms achieved sufficiently higher performances in their respective studies while being tested on a particular dataset. Consequently, it was confirmed that MMDF techniques outperform traditional baseline diagnostic approaches [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e][\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e]. Furthermore, the majority of the studies reviewed reported that the disease classification models achieved accuracy of more than 80% [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e][\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e][\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e][\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e]. A study using a DNN with two encoders that implement a multimodal fusion module with intra-modality self-attention and inter-modality cross-attention reported an accuracy of 76.8% [\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e]. Similarly, another study used in this review that used medical image analysis based on feature extraction, feature concatenation, and classification or diagnosis methods reported 99.94% accuracy in the classification or diagnosis of seven selected skin diseases. In general, as the analysis results show, MMDF techniques are significantly improving classification accuracies. Therefore, the utilization of multimodal data fusion techniques based on the deep learning methods, algorithms, and models in different settings (such as an ensemble of two or more of those methods, algorithms, and models) is a potential research area that needs further investigation, especially for the diagnosis of NTDs.\u003c/p\u003e \u003c/div\u003e"},{"header":"5. Discussion","content":"\u003cp\u003eThe primary goal of this systematic review is to collect and analyze research studies that are pertinent to the area of DL-based models that use multimodal data fusion techniques for the diagnosis of skin related NTDs. The analysis was conducted based on the guiding parameters initially set, which include the DL methods or approaches utilized for the diagnosis of the skin disease, the data fusion methods used, the type of medical data to be combined, the algorithms used, and the performance of each algorithm in the DL-based MMDF skin disease diagnostic model or system.\u003c/p\u003e \u003cdiv id=\"Sec23\" class=\"Section2\"\u003e \u003ch2\u003e5.1. Important Findings and Future Directions\u003c/h2\u003e \u003cp\u003eAfter collecting and analyzing 427 different articles, the nine articles used for the final review presented important dimensions in the area that include the fusion methods that can be used in different settings. In this regard, different fusion techniques were identified, such as image fusion [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e], feature fusion [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e][\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e][\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e], multimodal fusion for the combination of two or more modalities of data (such as two or more different modalities of image data) can be integrated with another modality of data, such as patient metadata [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e]. This clearly demonstrated that data fusion techniques can be used to integrate multiple types or modalities of data to enhance the performance of DL models, especially for the diagnosis of skin diseases. In this regard, as most of the NTDs can be diagnosed using multiple types of data coming from different sources, including skin lesions and patient metadata, multimodal data fusion can be a potential approach to be utilized for the diagnosis of NTDs.\u003c/p\u003e \u003cp\u003eA multimodal data fusion problem can also be approached from different perspectives. Some suggest a method for fusing data using deep CNN-based encoders and decoders for the extraction of image and metadata features to be combined, as well as using transformer modules [\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e][\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e]. Generally, the whole MMDF process can be implemented using four different DCNN modules that perform the data feature pre-processing, feature extraction, feature concatenation or combination, and finally disease classification. The very common task for all MMDF tasks is feature extraction, which will be used to extract features from the different modalities of data. Hence, proper feature extraction models and tools should be used for the feature extraction tasks. In deep learning, CNN-based pre-trained modes such as ResNet, VGG15/50, DenseNet, Inception, and similar DCNN architectures can be used for feature extraction.\u003c/p\u003e \u003cp\u003eRegarding algorithm utilization for the implementation of MMDF methods for the diagnosis of skin diseases, deep CNN models are by far the most experimented with and successful algorithms used. CNN, along with the DL-based CNN architectures, are playing important roles in the basic task of MMDF methods, feature extraction. Other algorithms, such as SVM, RF, and other ML algorithms, could also be used along with the CNN models. The implementation of MMDF methods using these algorithms achieved outperformance, as demonstrated by the studies analyzed in this review. Hence, as one of the potential beneficiaries, the diagnosis of NTDs can utilize these algorithms and methods to enhance the quality of diagnostic services.\u003c/p\u003e \u003cp\u003eAs potential feature work, the majority of the studies analyzed in this review share a common drawback, lack of larger datasets of images and metadata for training the intended models. It is crucial that a sufficient amount of quality and a balanced dataset containing the different modalities of data, including the different modalities of images and metadata, should be used to ensure the disease classification accuracy of the models to be developed.\u003c/p\u003e \u003cp\u003eFinally, from the strengths and weaknesses observed from the diverse studies analyzed in this systematic review, it was found that no single method of implementing the MMDF method guarantees a 100% achievement. However, with proper experimentation, analysis, and possible integration of one or more of the MMDF techniques presented in the analyzed studies, there is a high potential for enhancing the diagnostic quality of skin diseases. It is, therefore, worthwhile to experiment with and adapt the MMDF techniques for the diagnosis of NTDs, since the majority of NTDs are currently diagnosed using skin signs and symptoms with related patient metadata.\u003c/p\u003e \u003c/div\u003e"},{"header":"6. Conclusion","content":"\u003cp\u003eIn this systematic review, articles were collected from seven major and reputed sources where 427 study papers were organized, classified, screened and selected to analyze the application of DL-based diagnostic models using multimodal data fusion techniques for the diagnoses of skin related NTDs. Although there are studies that demonstrate the utilization of DL methods for the diagnoses of NTDs, no previous studies were found regarding the implementation of MMDF methods for the diagnoses of NTDs. Similar studies using MMDF for the diagnoses of other skin diseases, such as skin cancer, are reviewed to extract information about the implementation of these methods. In doing so, the selected studies are analyzed using parameters such as research approaches used, disease(s) selected for the study, the dataset used, algorithms used, the performance achieved, and future directions suggested by the study. Accordingly, although all the reviewed studies used diverse research methods and datasets based on their problem, DL-based CNN algorithms were found to be by far the most frequently used algorithm by all studies reviewed. In addition, DNN-based network architectures were widely utilized. In general, the implementation of MMDF methods for the diagnosis of skin diseases significantly enhances the diagnostic performances of models as per different studies reviewed, as confirmed in this review. Hence, utilizing MMDF methods for the diagnoses of skin diseases, particularly for skin related NTDs, would be paramount towards developing DL-based diagnostic models for NTDs.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eAll authors contributed equally to this work.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eWHO (2020) \u003cem\u003eEnding the neglect to attain the sustainable development goals: a road map for neglected tropical diseases 2021\u0026ndash;2030.\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePage W-Q (2023) accessed Jan. 15, Neglected tropical diseases. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.who.int/news-room/questions-and-answers/item/neglected-tropical-diseases\u003c/span\u003e\u003cspan address=\"https://www.who.int/news-room/questions-and-answers/item/neglected-tropical-diseases\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWorld Health Organization, Ending the neglect to attain the Sustainable Development Goals: A rationale for continued investment in tackling neglected tropical diseases 2021\u0026ndash;2030 (2022) [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://apps.who.int/iris/handle/10665/70809\u003c/span\u003e\u003cspan address=\"https://apps.who.int/iris/handle/10665/70809\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSouza AA et al (2021) Diagnostics and the neglected tropical diseases roadmap: Setting the agenda for 2030. Trans R Soc Trop Med Hyg 115(2):129\u0026ndash;135. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/trstmh/traa118\u003c/span\u003e\u003cspan address=\"10.1093/trstmh/traa118\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbdela SG et al (2020) Looking for NTDs in the skin; an entry door for offering patient centered holistic care. J Infect Dev Ctries 14 6.1, pp. 16S-21S. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3855/jidc.11707\u003c/span\u003e\u003cspan address=\"10.3855/jidc.11707\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAfza F, Sharif M, Khan MA, Tariq U, Yong HS, Cha J (2022) Multiclass Skin Lesion Classification Using Hybrid Deep Features Selection and Extreme Learning Machine. Sensors 22(3). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/s22030799\u003c/span\u003e\u003cspan address=\"10.3390/s22030799\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePacheco AGC, Krohling RA (2021) An Attention-Based Mechanism to Combine Images and Metadata in Deep Learning Models Applied to Skin Cancer Classification. IEEE J Biomed Heal Informatics 25(9):3554\u0026ndash;3563. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/JBHI.2021.3062002\u003c/span\u003e\u003cspan address=\"10.1109/JBHI.2021.3062002\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eH\u0026ouml;hn J et al (2021) Combining CNN-based histologic whole slide image analysis and patient data to improve skin cancer classification. Eur J Cancer 149:94\u0026ndash;101. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.ejca.2021.02.032\u003c/span\u003e\u003cspan address=\"10.1016/j.ejca.2021.02.032\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAnisuzzaman DM, Patel Y, Rostami B, Niezgoda J, Gopalakrishnan S, Yu Z (2022) Multi-modal wound classification using wound image and location by deep neural network. Sci Rep 12(1):1\u0026ndash;20. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/s41598-022-21813-0\u003c/span\u003e\u003cspan address=\"10.1038/s41598-022-21813-0\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003efeng Shen W, wei Tang H, Li Jbo, Li X, Chen S (2023) Multimodal data fusion for supervised learning-based identification of USP7 inhibitors: a systematic comparison. J Cheminform 15(1):1\u0026ndash;16. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1186/s13321-022-00675-8\u003c/span\u003e\u003cspan address=\"10.1186/s13321-022-00675-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGao J, Li P, Chen Z, Zhang J (2020) A survey on deep learning for multimodal data fusion. Neural Comput 32(5):829\u0026ndash;864. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1162/neco_a_01273\u003c/span\u003e\u003cspan address=\"10.1162/neco_a_01273\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLipkova J et al (2022) Artificial intelligence for multimodal data integration in oncology. Cancer Cell 40(10):1095\u0026ndash;1110. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.ccell.2022.09.012\u003c/span\u003e\u003cspan address=\"10.1016/j.ccell.2022.09.012\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e[13] N, Luo X, Zhong L, Su Z, Cheng W, Ma, Hao P (2023) Artificial intelligence-assisted dermatology diagnosis: From unimodal to multimodal. Comput Biol Med 165:107413. no. July10.1016/j.compbiomed.2023.107413\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBeesetty R et al (2023) Leprosy Skin Lesion Detection: An AI Approach Using Few Shot Learning in a Small Clinical Dataset, \u003cem\u003eIndian J Lepr\u003c/em\u003e, vol. pp. 89\u0026ndash;102, 2023, [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.ijl.org.in\u003c/span\u003e\u003cspan address=\"http://www.ijl.org.in\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAli Z et al (2022) A Proposed Framework for Early Prediction of Schistosomiasis. Diagnostics 12(12):1\u0026ndash;25. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/diagnostics12123138\u003c/span\u003e\u003cspan address=\"10.3390/diagnostics12123138\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSteyve N, Steve P, Ghislain M, Ndjakomo S, pierre E (September, 2022) Optimized real-time diagnosis of neglected tropical diseases by automatic recognition of skin lesions. Inf Med Unlocked 33. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.imu.2022.101078\u003c/span\u003e\u003cspan address=\"10.1016/j.imu.2022.101078\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYotsu RR, Ding Z, Hamm J, Blanton RE (2023) Deep learning for AI-based diagnosis of skin-related neglected tropical diseases: A pilot study. PLoS Negl Trop Dis 17(8):1\u0026ndash;12. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1371/journal.pntd.0011230\u003c/span\u003e\u003cspan address=\"10.1371/journal.pntd.0011230\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCastanedo F (2013) A review of data fusion techniques, \u003cem\u003eSci. World J.\u003c/em\u003e, vol. 2013, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1155/2013/704504\u003c/span\u003e\u003cspan address=\"10.1155/2013/704504\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShoumy NJ, Ang LM, Seng KP, Rahaman DMM, Zia T (2020) Multimodal big data affective analytics: A comprehensive survey using text, audio, visual and physiological signals. J Netw Comput Appl 149:102447. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jnca.2019.102447\u003c/span\u003e\u003cspan address=\"10.1016/j.jnca.2019.102447\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang T et al (2021) A Feature Fusion Method with Guided Training for Classification Tasks, \u003cem\u003eComput. Intell. Neurosci.\u003c/em\u003e, vol. no. c, 2021, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1155/2021/6647220\u003c/span\u003e\u003cspan address=\"10.1155/2021/6647220\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLu X, Duan X, Mao X, Li Y, Zhang X (2017) Feature Extraction and Fusion Using Deep Convolutional Neural Networks for Face Detection, \u003cem\u003eMath. Probl. Eng.\u003c/em\u003e, vol. 2017, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1155/2017/1376726\u003c/span\u003e\u003cspan address=\"10.1155/2017/1376726\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlDahoul N, Karim HA, Ba Wazir AS (2021) Model fusion of deep neural networks for anomaly detection. J Big Data 8(1). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1186/s40537-021-00496-w\u003c/span\u003e\u003cspan address=\"10.1186/s40537-021-00496-w\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKaur H, Koundal D, Kadyan V (2021) Image Fusion Techniques: A Survey. Arch Comput Methods Eng 28(7):4425\u0026ndash;4447. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s11831-021-09540-7\u003c/span\u003e\u003cspan address=\"10.1007/s11831-021-09540-7\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang Y, Cai J, Louie DC, Wang ZJ, Lee TK, June (2021) 104812 doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.compbiomed.2021.104812\u003c/span\u003e\u003cspan address=\"10.1016/j.compbiomed.2021.104812\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAn I, Harman M, Ibiloglu I (2017) Topical Ciclopirox Olamine 1%: Revisiting a Unique Antifungal. Indian Dermatol Online J 10(4):481\u0026ndash;485. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.4103/idoj.IDOJ\u003c/span\u003e\u003cspan address=\"10.4103/idoj.IDOJ\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eY\u0026eacute;lamos O, Mary Diem L, Braun RP, French KK, Marghoob AA (2019) Dermoscopy for Dermatopathologists. Elsevier Inc.. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/B978-0-323-37457-6.00028-6\u003c/span\u003e\u003cspan address=\"10.1016/B978-0-323-37457-6.00028-6\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePawłowski M, Wr\u0026oacute;blewska A, Sysko-Romańczuk S (2023) Sensors 23(5):1\u0026ndash;16. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/s23052381\u003c/span\u003e\u003cspan address=\"10.3390/s23052381\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Effective Techniques for Multimodal Data Fusion: A Comparative Analysis\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePage MJ et al (2021) The prisma 2020 statement: An updated guideline for reporting systematic reviews. Med Flum 57(4):444\u0026ndash;465. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.21860/medflum2021_264903\u003c/span\u003e\u003cspan address=\"10.21860/medflum2021_264903\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePRISMA, PRISMA (2023) : TRANSPARENT REPORTING of SYSTEMATIC REVIEWS and META-ANALYSES. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.prisma-statement.org/\u003c/span\u003e\u003cspan address=\"http://www.prisma-statement.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (accessed Nov 15,\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAI R (2023) accessed Oct. 20, RAYYAN. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.rayyan.ai\u003c/span\u003e\u003cspan address=\"https://www.rayyan.ai\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e/\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBakkouri I, Afdel K (2020) Computer-aided diagnosis (CAD) system based on multi-layer feature fusion network for skin lesion recognition in dermoscopy images. Multimed Tools Appl 79:29\u0026ndash;30. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s11042-019-07988-1\u003c/span\u003e\u003cspan address=\"10.1007/s11042-019-07988-1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMahum R, Aladhadh S (2022) Skin Lesion Detection Using Hand-Crafted and DL-Based Features Fusion and LSTM, \u003cem\u003eDiagnostics\u003c/em\u003e, vol. 12, no. 12, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/diagnostics12122974\u003c/span\u003e\u003cspan address=\"10.3390/diagnostics12122974\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang D, Li A, Wu W, Yu L, Kang X, Huo X (2023) CR-Conformer: a fusion network for clinical skin lesion classification. Med Biol Eng Comput 012345678910.1007/s11517-023-02904-0\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAfra S, Alhajj R (2019) al Pr p ro of. Phys A 123151. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jpi.2023.100341\u003c/span\u003e\u003cspan address=\"10.1016/j.jpi.2023.100341\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYap J, Yolland W, Tschandl P (2018) Multimodal skin lesion classification using deep learning. Exp Dermatol 27(11):1261\u0026ndash;1267. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1111/exd.13777\u003c/span\u003e\u003cspan address=\"10.1111/exd.13777\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi W, Zhuang J, Wang R, Zhang J, FUSING METADATA AND DERMOSCOPY IMAGES FOR SKIN, DISEASE DIAGNOSIS School of Data and Computer Science (2020) Sun Yat-sen University, China Key Laboratory of Machine Intelligence and Advanced Computing, MOE, Guangzhou. China Department of Computer Science an, pp 1996\u0026ndash;2000\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOu C et al (2022) A deep learning based multimodal fusion model for skin lesion diagnosis using smartphone collected clinical images and metadata, \u003cem\u003eFront. Surg.\u003c/em\u003e, vol. 9, no. October, pp. 1\u0026ndash;9, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3389/fsurg.2022.1029991\u003c/span\u003e\u003cspan address=\"10.3389/fsurg.2022.1029991\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCai G, Zhu Y, Wu Y, Jiang X, Ye J, Yang D (2023) A multimodal transformer to fuse images and metadata for skin disease classification. Vis Comput 39(7):2781\u0026ndash;2793. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s00371-022-02492-4\u003c/span\u003e\u003cspan address=\"10.1007/s00371-022-02492-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlmuayqil SN, Abd El-Ghany S, Elmogy M (2022) Computer-Aided Diagnosis for Early Signs of Skin Diseases Using Multi Types Feature Fusion Based on a Hybrid Deep Learning Model. Electron 11(23). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/electronics11234009\u003c/span\u003e\u003cspan address=\"10.3390/electronics11234009\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen Q, Li M, Chen C, Zhou P, Lv X, Chen C (2023) MDFNet: application of multimodal fusion method based on skin image and clinical data to skin cancer classification. J Cancer Res Clin Oncol 149(7):3287\u0026ndash;3299. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s00432-022-04180-1\u003c/span\u003e\u003cspan address=\"10.1007/s00432-022-04180-1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003eTable 3 is available in the Supplementary Files section.\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Deep Learning, Disease Diagnosis, Multimodal Data Fusion, Skin NTDs","lastPublishedDoi":"10.21203/rs.3.rs-3870993/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3870993/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eNeglected tropical diseases (NTDs) are the most prevalent diseases worldwide, affecting one-tenth of the world population. Although there are multiple approaches to diagnosing these diseases, using skin manifestations and lesions caused as a result of these diseases along with other medical records is the preferred method. This fact triggers the need to explore and implement a deep learning-based diagnostic model using multimodal data fusion (MMDF) techniques to enhance the diagnostic process. This paper, thus, endeavors to present a thorough systematic review of studies regarding the implementation of MMDF techniques for the diagnosis of skin-related NTDs. To achieve its objective, the study used the PRISMA method based on predefined questions and collected 427 articles from seven major and reputed sources and critically appraised each article. Since no previous studies were found regarding the implementation of MMDF for the diagnoses of skin related NTDs, similar studies using MMDF for the diagnoses of other skin diseases, such as skin cancer, were collected and analyzed in this review to extract information about the implementation of these methods. In doing so, various studies are analyzed using six different parameters, including research approaches, disease selected for diagnosis, dataset, algorithms, performance achievements, and future directions. Accordingly, although all the studies used diverse research methods and datasets based on their problems, deep learning-based convolutional neural networks (CNN) algorithms are found to be the most frequently used and best-performing models in all the studies reviewed.\u003c/p\u003e","manuscriptTitle":"Applying Multimodal Data Fusion based on Deep Learning Methods for the Diagnosis of Neglected Tropical Diseases: A Systematic Review","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-01-19 16:39:48","doi":"10.21203/rs.3.rs-3870993/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"1f6b201c-eca7-43a3-a89e-8f6c7130e384","owner":[],"postedDate":"January 19th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-02-04T23:31:49+00:00","versionOfRecord":[],"versionCreatedAt":"2024-01-19 16:39:48","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3870993","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3870993","identity":"rs-3870993","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}