{"paper_id":"e03b5e29-adcf-474f-aa65-7e71b581b539","body_text":"The current COVID-19 pandemic has impacted the world with over 18.35 million\ninfections and over 6,96,147 deaths so far (as of  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$5^{\\text {th}}$ \\end{document}  August\n2020)  [1] . Early identifying, isolation and\ncare for patients is a key strategy for a better management of this pandemic. Our\nstudy aims to provide a conceptual transfer learning framework to support COVID-19\ndetection with the use of image classification using deep learning models for\nmultiple imaging modes including X-Ray, Ultrasound, and CT scan. The acquisition of\na sufficiently large, publicly available corpus of medical image sample data for\nfully training deep learning models is challenging for novel medical conditions such\nas COVID-19 since collection and labelling of images requires significant time and\nresources to compile. An alternative method of training deep learning models is\n“transfer learning” whereby a deep learning network is pre-weighted\nwith the results of a previous training cycle from a different domain. This\ntechnique is commonly used as a basis for initializing deep learning models which\nare then fine-tuned using the limited available medical sample data set with results\nthat have been documented to outperform fully trained networks under certain\ncircumstances  [2] ,  [3] . The study will demonstrate how transfer learning can be\nused for COVID-19 detection for three commonly used imaging modes X-Ray, Ultrasound,\nand CT scan. This could assist practitioners and researchers in developing a\nsupporting tool for highly constrained health professionals in determining the\ncourse of treatment. The study further demonstrates a pre-processing pipeline for\nimproving the image quality, for deep learning-based predictions. An initial testing\nis also conducted to understand the suitability of various popular deep learning\nmodels for the limited available dataset in order to select a model for the proposed\nimage classification demonstrations on multiple image modes.\nFast, accessible, affordable and reliable identification of COVID-19 pathology in an\nindividual is key to slowing the transmission of COVID-19 infection. Currently,\nreverse transcriptase quantitative polymerase chain reaction (RT-qPCR) tests are the\ngold standard for diagnosing COVID-19  [4] .\nDuring this test small amounts of viral RNA are extracted from a nasal swab,\namplified, and quantified with virus detection indicated visually using a\nfluorescent dye. Unfortunately, the RT-qPCR test is manual and time-consuming, with\nresults taking up to two days. Some studies have also shown false positive\nPolymerase Chain Reaction PCR testing  [5] .\nOther testing approaches include imaging technology-based approaches including\ncomputed tomography (CT) imaging  [6]  and\nX-Ray imaging based  [7] ,  [8]  and Ultrasound imaging  [9] .\nThe CT scan-based COVID-19 detection is time consuming and manual with the\nrequirement of expert involvements. CT scanning machines are also difficult to use\nfor COVID patients, as the patients often need to be transferred to the CT room, the\nmachines would require extensive cleaning after each usage, and higher radiation\nrisks  [10] . Although CT is not recommended\nas a primary diagnostic tool, it has been successfully used as a supporting tool for\nCOVID-19 condition assessment  [6] . Common CT\nfindings include ground-glass opacities (GGO) at the early stage, during progressive\nstage, air space consolidation during peak stage, Broncho vascular thickening in the\nlesion, and traction bronchiectasis are visible during absorption stage  [10] . Several studies have shown promising\nresults in using deep learning models to automated diagnosis of COVID-19 from CT\nimages  [6] ,  [11] ,  [12] . Both the\nPCR tests and CT scans are comparatively costly  [13] ,  [14]  and with an\noverwhelming demand many countries are forced to perform selective testing for only\nhigh-risk population.\nX-Ray imaging is relatively cost effective and commonly utilized for lung infection\ndetection and is useful for COVID-19 detection as well  [15] . Medical observations were made by one of the co-authors\nof this research (Dr. Saha) who is also a medical professional, as well as by\ntreating doctors of the COVID-19 dataset  [16]  patients. The common features observed in the X-Ray images of\npatients with COVID-19 are patchy infiltrates or opacities that bear similarities to\nother viral pneumonia features. X-Ray images do not show any abnormalities in the\nearly stages of COVID-19. However, as the disease progresses, COVID-19 gradually\nmanifests as a typical unilateral patchy infiltration involving mid zone and upper\nor lower zone of the lungs, occasionally with evidence of a consolidation.\nUltrasound imaging has also been recommended as a tool for COVID-19 lung condition\nassessment since it can be used at bedside with minimal infection spreading risks\nand has excellent ability to detect lung conditions related to COVID-19  [17] . Progression of COVID-19 infection is\nevident as B-lines aeration in early stages of consolidation in critical stages\n [10] ,  [18] .\nFig. 1  shows the progression of evidence for\nthe patient in the COVID-19 datasets for X-Ray, CT and Ultrasound imaging.  FIGURE 1. COVID-19 progression over several days as evident in different imaging\nmodes.\nCOVID-19 progression over several days as evident in different imaging\nmodes.\nComputer vision diagnostic tools for COVID-19 from multiple imaging modes such as\nX-Ray, Ultrasound, and CT would provide an automated “second reading”\nto clinicians, assisting in the diagnosis and criticality assessment of COVID-19\npatients to assist in better decision making in the global fight against the\ndisease. COVID-19 often results in pneumonia, and for radiologists and practitioners\ndifferentiating between the COVID-19 pneumonia and other types of pneumonia (viral\nand bacterial) solely based on diagnostic images could be challenging  [19] .\nDeep learning artificial neural networks, and the Convolutional Neural Networks\n(CNNs) have proven to be highly effective in a vast range of medical image\nclassification applications  [20] ,  [21] . In this study, we present three key\ncontributions. Primarily, we demonstrate how transfer learning capabilities of off\nthe shelf deep learning models can be utilized to perform classifications in two\ndistinct scenarios for three imaging modes X-Ray, Ultrasound, and CT scan:  1) Identifying the pneumonia (both COVID-19 and other types) affected lung\nagainst the normal lung. 2) Identifying COVID-19 affected lung from non COVID-19 pneumonia affected\nlung.\nIdentifying the pneumonia (both COVID-19 and other types) affected lung\nagainst the normal lung.\nIdentifying COVID-19 affected lung from non COVID-19 pneumonia affected\nlung.\nSecondly, we present a comparative study in order to select a suitable deep learning\nmodel for our demonstration. We performed a comparative testing of several common\noff-the-shelf CNN models namely VGG16/VGG19  [22] , Resnet50  [23] , Inception\nV3  [24] , Xception  [25] , InceptionResNet  [26] , DenseNet  [27] , and\nNASNetLarge  [28] . The testing is not\nintended for exhaustive performance comparison of these methods, rather we wanted to\nselect the most suitable one for our multi-modal image classification, which\nperforms decently with minimal tuning. The source X-Ray, Ultrasound, and CT image\nsamples, especially those from the COVID-19 data sets, have been harvested from\nmultiple sources and are of inconsistent quality. In our final contribution, we have\nimplemented a pre-processing pipeline to reduce unwanted signal noise such as\nnon-lung area visible in X-Rays, and thereby reduce the impact of sampling bias on\nthis comparison. Through this pre-processing pipeline, we minimize the image quality\nimbalances in the image samples. This would allow models to train on lung features\nonly thus having a greater chance of learning disease features and ignoring other\nnoise features. The study would provide timely model selection guidelines to the\npractitioners who often are resorted to utilise certain mode of imaging due to time\nand resource scarcity.\nIn the following sections we first present a brief review of recent scholarly works\nrelated to this study, followed by a discussion on the datasets we used and related\nchallenges. We then present the dataset generation process along with our proposed\npre-processing pipeline for data quality balancing. We then present the deep\nlearning model selection process along with comparison results. Finally, we present\nthe performance results with discussions for our selected model on all three image\nmodes.\n\nComputer aided detection and diagnosis of pulmonary pathologies from X-Ray images is\na field of research that started in the 1960s and steadily progressed in the\nfollowing decades with papers describing highly accurate diagnosis of a range of\nconditions including osteoporosis  [29] ,\nbreast cancer  [30] , and cardiac disease\n [31] .\nCT scans also use X-Rays as a radiation source, however, they provide much higher\nimage resolution and contrast compared to standard X-Ray images because of a much\nmore focused X-Ray beam used to produce cross-sectional images of the patient  [32] . CT is generally considered as the best\nimaging modality for lung parenchyma and is widely accepted by clinicians as the\n“gold standard”  [33] . A large\ncorpus of research exists relating to the use of machine learning to improve the\nefficiency and accuracy of lung cancer diagnosis – largely driven by\nextensive CT based lung cancer screening programs in many parts of the world.\nSeveral researches have achieved incredibly accurate results using CNNs with\ntransfer learning to detect lung nodules  [34] – [37] . Recently a deep\nlearning system built by Google achieved state-of-the-art performance using\npatients’ current and prior CT volumes to predict the risk of lung cancer.\nThis system outperformed human radiologists where prior CT scans were not available,\nand equaled human radiologist performance where historical CT scans were available\n [38] . Although X-Ray is the current\nreference diagnosis for pneumonia, some studies point out that CT generally\noutperforms X-Ray as a diagnostic tool for pneumonia, albeit at higher cost and\nconvenience  [39] ,  [40] .\nUltrasound has traditionally been used diagnostically in the fields of cardiology and\nobstetrics and more recently for a range of other conditions covering most organs.\nOne of the reasons for this increase in the use of ultrasound is that technical\nadvancements including machine learning have allowed useful information to be\ndetermined from the low quality and high signal-to-noise images that are typical of\nthe Ultrasound imaging modality  [41] .\nSeveral researchers have recently used Ultrasound as an effective diagnostic aid in\nhepatic steatosis, adenomyosis, and craniosynostosis  [42] , Pancreatic cancer  [43] , Breast cancer  [44]  and\nprostate cancer  [45] . Use of bedside\nultrasound in critically ill patients compared favorably against chest X-Ray and\napproached the diagnostic accuracy of CT scans for a range of thoracic conditions\n [46] . The combination of lung\nultrasound with machine learning techniques was found to be valuable in providing\nfaster and more accurate bedside interpretation of lung ultrasound for acute\nrespiratory failure  [47] .\nDifficulties in distinguishing soft tissue caused by poor contrast in X-Ray images\nhave led some researchers to implement contrast enhancement  [48]  as a pre-processing step in X-Ray based diagnosis. In\naddition, lung segmentation of X-Ray images is an important step in the\nidentification of lung nodules and various segmentation approaches are proposed in\nthe literature based on linear filtering/thresholding, rolling ball filters and more\nrecently CNNs  [49] .\nAlthough CT scans are much higher contrast/resolution compared to X-Ray factors such\nas low dose and improper use of image enhancement can lead to poor quality images. A\nnumber of researchers have noted that histogram equalization techniques,\nparticularly adaptive histogram equalization can improve the contrast of CT images\n [50] . A combination of histogram\nnormalization, gamma correction and contrast limited adaptive histogram equalization\nhas been shown to objectively improve the quality of poor contrast CT images  [51] .\nUltrasound images tend to be noisy due to the relatively low penetration of\nsoundwaves into organic tissue compared to X-Rays. This limitation has led a number\nof researchers to develop methods to improve the quality of ultrasound images by\nvarious means including noise filtering, wavelet transformation and deconvolution\n [52] . Contrast Limited Adaptive\nHistogram Equalization (CLAHE) has been used as part of a pre-processing pipeline to\nenhance the quality of ultrasound images  [53] .\nIn the literature review we noted a small number of very recent studies that have\nused deep learning systems for COVID-19 screening and diagnosis. A custom-built\n18-layer residual network pre-trained on the ImageNet weights against COVID-19 (100\nimages) and Pneumonia (1431 images) X-Ray image datasets  [54] . A range of deep learning frameworks coined as\nCOVIDX-Net trained on a small data set of 25 confirmed COVID-19 cases  [55] . A custom curated dataset of COVID-19,\nviral pneumonia and normal X-Ray images  [56] . A custom residual CNN that was highly effective in distinguishing\nbetween COVID-19, Pneumonia and normal condition X-Ray images  [57] . These studies used the COVID-19 dataset  [16]  for the COVID-19 X-Ray samples, and the\nRSNA dataset  [58]  was used to get pneumonia\nand normal X-Ray samples.\nAutomated COVID-19 Pneumonia diagnosis from CT scans has been the focus of recent\nstudies with promising results  [59] – [62] . A combined U-Net\nsegmentation and 3D classification CNN has been used to accurately predict the\npresence of COVID-19 with an accuracy of 90% using a non-public dataset of CT\nimages  [63] . A ResNet50 based CNN with\ntransfer learning from the ImageNet weights was able to classify COVID-19 with\n94% accuracy  [64]  against a normal\ncondition CT slice using unspecified multiple international datasets as a corpus. In\na recent work,  [65]  addressed the challenge\nof automatically distinguishing between COVID-19 and community acquired pneumonia\nusing machine learning. This system uses a U-Net pre-processor model for lung field\nsegmentation, followed by a 3D ResNet50 model using transferred ImageNet weights.\nThis study achieved a sensitivity of 87% against a large non-public dataset\ncollected from 6 hospitals. The DenseNet-169 CNN has been used  [66]  to detect COVID-19 vs non-COVID-19 CT slices. Without\nsegmentation this system achieved an accuracy of 79.5% with an F1 score of\n76%. Using joint segmentation, the classification accuracy was raised to\n83.3% with an F1 score of 84.6%.\nThere has been less attention given to the use of machine learning to automate\nCOVID-19 diagnosis from Ultrasound images, however a ResNet based CNN trained on the\navailable Ultrasound COVID-19 data has achieved an accuracy of 89% with\nrecall accuracy for COVID-19 of 96%  [9] .\nEach imaging mode differs in terms of cost/availability and the level of clinical\nexpertise required to accurately interpret the generated medical images. Different\nimaging modes are therefore suitable to different contexts – for example both\nX-Ray and Ultrasound can be implemented as low-cost portable units that may be used\nas bedside or even as field diagnostic tools. CT scanning equipment is typically\nphysically fixed at high cost and is therefore only available within the confines of\nhospitals and medical clinics. Our main aim is to first select one suitable deep\nlearning model through comparative testing of a range of off-the-shelf deep learning\nmodels against each of these imaging modes using transfer learning. The comparison\nresults are then used to address limited sample data size and data variability. We\nthen applied image pre-preprocessing to improve image quality and reduce inter and\nintra dataset systematic differences in brightness and contrast level. Finally, we\nperformed extensive parameter tuning on the selected model and compared the\nperformance of this model for each imaging mode.\n\nLarge numbers of X-Ray, CT and Ultrasound images are available from several\npublicly accessible datasets. With the emergence of COVID-19 being very recent\nnone of these large repositories contain any COVID-19 labelled data, thereby\nrequiring that we rely upon multiple datasets for Normal, Pneumonia, COVID-19\nand other non COVID-19 source images.\nCOVID-19 chest X-Rays were obtained from the publicly accessible COVID-19 Image\nData Collection  [16] . This collection\nhas been sourced from websites and pdf format publications. Unsurprisingly, the\nimages from this collection are of variable size and quality. Image contrast\nlevels, brightness and subject positioning are all highly variable within this\ndataset. Our analysis in this article is based on a download of this dataset\nmade on 11 May 2020.\nThe selection of a dataset for Normal and Pneumonia condition X-Rays posed a\ndilemma since a highly curated data set is not comparable to the available\nCOVID-19 chest X-Ray dataset. Our early tests against one such dataset gave an\nunrealistically high classification accuracy for the quality of the data under\ntest. We found that the National Institute of Health (NIH) Chest X-Ray  [67]  dataset – provided images are\nof a similar size,  [68]  quality and\naspect ratio to the typical images in the COVID-19 dataset with dimensions being\nuniformly  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$1024 \\times 1024$ \\end{document} \npixels in a portrait orientation.\nCT scans for COVID-19 and non COVID-19 were obtained from the publicly accessible\nCOVID-CT Dataset  [66] . This dataset has\nbeen sourced by extracting CT slice images showing the COVID-19 pathology from\npreprint papers. Once again, the images from this collection are of variable\nsize and quality. Moreover, the process of CT scanning is dynamic, with a full\nscan consisting of many discrete slices taken in a helical pattern along the\nthoracic cavity. The images in this collection only present a single, or small\nnumber of slices per patient. As CT slice images progress along the thoracic\ncavity, the structural features visible in the generated image change\ndramatically. Ideally all slices would be available for analysis in order to\nequalize the distribution/prominence in the image of these features, however\nthis is not the case with this dataset. Our analysis in this article is based on\na download of this dataset made on 11 May 2020.\nUltrasound images for COVID-19, Pneumonia and Normal conditions were obtained\nfrom the publicly accessible POCOVID-Net data set  [9] . These images have been sampled from video sourced\nfrom various online sources. We noticed a huge variation in the quality in the\nimages within each condition caused by the position of the ultrasound apparatus\non the patients’ chest. Ideally ultrasound video would be taken in a\nsystematic way to allow for greater comparability of the condition datasets with\nevery frame in the video subject to analysis. Neither of these conditions are\nsatisfied by this dataset. Our analysis in this article is based on a download\nof this dataset made on 11 May 2020.\nThe number of images of each dataset along with a description of the\ncharacteristics of the datasets is described in  Table 1 . We believe the significant quality variations\nbetween data from different classes need to be balanced for deep learning models\nto learn actual disease related variations. Therefore, our study stresses the\nimportance of sampling bias/signal noise removal from the image datasets prior\nto using them for model development and classification in order to obtain\nmeaningful and trustworthy classification results. Some illustrative examples of\nthis variability of these datasets is shown in  Fig. 2 . Of these examples images (b), (c) and (f) appear to have\nbeen cropped from journal articles and in the case of (f) scanned. These images\nare of poor quality and lacking detail that would indicate a pathology to our\nmachine learning models. Images (g), (j) and (k) also lack detail as a result of\napparatus positioning. Images (d) and (l) show high brightness and low contrast,\nthus hiding pathological details. Despite the variability of the datasets we\nchose to only very lightly curate data as described in Section III(C) Data\nPre-processing and shown in  Table 2 .\nOur reasoning for this is twofold. Firstly we wish to avoid biasing the data\ncorpus with a non-expert subjective opinion of pathological indications, and\nsecondly we consider the usefulness of this study to potentially extend to\nfuture pandemic situations where similar data quality issues will be likely if\nnot inevitable. TABLE 1 Summary of Data Sources Used Collection Number of Images Characteristics Notes COVID-19 Image Data Collection  [16] 115: COVID-19 (PA) Variable size, quality, contrast and\nbrightness. Only source of publicly accessible\nCOVID-19 PA X-Ray images and used in this study. NIH Chest X-Ray  [67] 322: Pneumonia 60361: No\nFinding Intra-dataset uniformity similar to\nCOVID-19 dataset. All images are 1024 x1024 in size. Objectively similar in quality to\nthe COVID-19 Image Data Collection. Used in this study. COVID-CT Dataset  [66] 349: COVID-19 397: Non COVID Variable size, contrast and\nbrightness Only source of publicly accessible\nCOVID-19 CT images and used in this study. POCOVID-Net Dataset  [9] 654: COVID-19 277: Pneumonia 172: No\nFinding Variable size, contrast and\nbrightness Only source of publicly accessible\nCOVID-19 Ultrasound images and used in this study. TABLE 2 Sampled Dataset for Experiments Image Mode Condition Source images Curated images Number of augmented images Source X-Ray COVID-19 140 139 2,920 [16] Pneumonia 322 190 2,920 [67] Normal 60361 400 5,840 [67] CT COVID-19 349 349 6,000 [66] Non COVID 397 397 6,000 [66] Ultrasound COVID-19 399 399 2,720 [9] Pneumonia 277 277 2,720 [9] Normal 235 235 5,440 [9] \n FIGURE 2. Different variations observed in the COVID-19 datasets.\nDifferent variations observed in the COVID-19 datasets.\nIn this study, we aim to use real X-Ray, Ultrasound and CT scan data only, and\nnot considering creation and use of synthetic data at this stage. We also used a\nrelatively balanced dataset size for our model experiments, with imbalance\naddressed using calculated class training weights. From the source datasets\nshown in  Table 1 , we created a master\ndataset for our experiments.\nThe X-Ray COVID-19 dataset was lightly curated to remove a single image that was\nan incorrectly labelled projection. All other COVID-19 images were included for\nthe various modes. The non-COVID-19 images were also lightly curated to remove\nimages that were mislabeled projections or dominated by intrusive medical\ndevices. This left us with usable image samples for X-Ray, Ultrasound and CT\nscan to work with. Since the resulting sample corpus was still relatively small\nfor a deep learning application, we applied several data augmentation\ntransformations including horizontal flip, horizontal and vertical shift and\nrotation during the experiment process to increase the volume and variety of the\nsample set as summarized in  Table\n2 .\nWe recognized that although we were using only a small number of image\ncollections for our experiments; we were in fact relying upon data sourced from\nan unknown number and variety of X-Ray, CT, and Ultrasound machines, each with\nvariable exposure parameters and operator behaviors. Systematic image exposure\nand brightness differences within and between the datasets of each imaging mode\nproved to be particularly concerning, and several researchers have indicated\nthat medical image analysis methods are highly sensitive to variations of image\nintensities  [69] . Research has shown\nthat the feasibility of an automated analysis system requires that “the\nintensity of the same tissue should always be similar in one image and not vary\nwith the location of the tissue”  [69] . This principle, when extrapolated to the many images utilized\nin machine learning algorithms, implies that all images in the sample data sets\nshould have similar intensity for the same tissue over the entire set of images\nas well as within a single image.\nSince the machine learning classifiers, use a pixel array as a data source, any\nsystematic difference in pixel intensity between the datasets would introduce\nsampling bias in the results. This would have the consequence of training the\nmachine learning classifiers on systematic image histogram differences rather\nthan the actual clinical image content of interest.\nTo minimize the effect of sampling bias, we applied histogram equalization to\nimages using the N-CLAHE method described by  [70] . The method both normalizes images and enhances small details,\ntextures and local contrast by first globally normalizing the image histogram\nfollowed by application of Contrast Limited Adaptive Histogram Equalization\n(CLAHE)  [71] . This was implemented\nusing the OpenCV equalizeHist and createCLAHE functions  [72] .\nAs shown in  Fig. 3 , the N-CLAHE\npre-process greatly improved the image brightness/contrast consistency across\nthe datasets as well as enhancing the fine details of these images. The same\neffect was observed within each data set. Subjectively, the authors can no\nlonger easily tell which image has been drawn from which dataset purely on image\nbrightness and contrast characteristics alone.  FIGURE 3. Results of enhancement preprocessing on original samples for\nCOVID-19, Pneumonia and Normal images.\nResults of enhancement preprocessing on original samples for\nCOVID-19, Pneumonia and Normal images.\n\nThe experiment pipeline is shown in  Fig.\n4 . Unprocessed images are read with directory names used as class labels.\nN-CLAHE is then applied to normalize images and highlight the finer details for\nthe attention of the machine learning classifiers. Images are then resized to\nthe classifier default size, for example  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$224\\times 224$ \\end{document} \npixels for VGG16/19 and  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$299\\times 299$ \\end{document} \npixels for InceptionV3. Following image resizing, data augmentation is applied\nto increase the number and variation of images provided to the classifier.\nAugmentations applied include horizontal flip, rotation, width shift, and height\nshift. Vertical flip was not applied since X-Ray images are not vertically\nsymmetrical, and the resulting flipped image would not resemble a real chest\nX-Ray. Finally, the augmented images are utilized by the machine learning\nclassifier using an 80:20 Train/Test split.  FIGURE 4. Experiment pipeline for preprocessing and classification.\nExperiment pipeline for preprocessing and classification.\nOne of the key objectives of this study was to achieve reliable classification\nresults using publicly accessible data and “out-of-the-box” models\nwith transfer learning to both compensate for the limited size of the sample\ndata sets, and to accelerate the training process so that this could be\nreasonably performed on modest hardware.\nCNN based models are well utilized for image classification purposes and we want\nto initially select a suitable CNN based deep learning model for our multimodal\nimage classification study. Our primary aim is not to perform exhaustive\nperformance evaluation among all available models, rather we aim to show the\ngeneric applicability of popular model genres for the challenging and limited\ntime critical dataset for COVID-19 chest images in multiple modes including\nX-Ray, CT and Ultrasound to provide reasonable precision.\nSince the ImageNet challenge (ILSVRC) 2012  [73] , there has been a flurry of development in deep learning models\nfor image classification and computer vision applications. These developed\nmodels can be broadly grouped into several distinct model families such as\nAlexNet, VGG Nets, Inception Nets, ResNets, MobileNets, DenseNets and NASNets\nbased on their distinct architectures  [74] ,  [75] . Over the years,\nthese basic model families produced several versions  [74]  and they have been extensively used by other\nresearchers to develop modified and hybrid models  [76] . Recent studies attempted to improve performance of\nthe base models by proposing new layers and filters such as Sparse Shift Filter\n [77] , Asymmetric Convolution Block\n [78] , Adder Networks  [79] , Virtual Pooling  [80] , Discrete Wavelet Transform  [81] , and HetConv  [82] , etc. Some recent substantial models have been\ndeveloped based on the base models, such as Res2Net  [83]  and Wide ResNet  [84]  using the ResNet model; while Log Dense Net  [85]  and Sparse Net  [86]  using the DenseNet model. Another path of\ndevelopment combining multiple base models has resulted in a number of hybrid\nmodels including AOGNet  [87] , PNASNet\n [88] , AmoebaNet  [89] , DPN  [90] , HCGNet  [76] , GCNet  [91] , ThiNet\n [92] , and SKNet  [93]  etc.\nIn order to find the most suitable model for our study, we focused on widely\npopular models, suitable for transfer learning, and readily available in\npackaged form through trusted public libraries such as Keras. Hence, we only\nconsidered representatives of the base models in this domain as discussed below.\nConveniently, these models are all available as part of the Keras API and each\nsupport transfer learning  [94]  in the\nform of supporting the pre-application to the model of the ImageNet  [95]  weights.\nVGG16 and VGG19  [22]  are\nconvolutional neural network (CNN) architectures with very small convolution\nfilters ( \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$3\\times 3$ \\end{document} ) and\na stride of 1 designed to achieve high accuracy in large-scale image\nrecognition applications. The two implementations differ in depth of\nconvolution/max-pooling and fully connected layers, with VGG16 having 16\nlayers in the base model and VGG19 having 19 layers.\nThe ResNet  [23]  CNN was developed as\na means of avoiding the vanishing gradient problem inherent in deep neural\nnetworks by implementing a system of skip connections between layers\n– known as residual learning. This architecture results in a network\nthat is more efficient to train, allowing for deeper networks to be designed\nthat positively impact the model accuracy. ResNet50 is such a network with\n50 layers of implementing residual learning.\nThe Inception V3  [24]  CNN aimed to\nimprove utilization of computing resources inside the network by increasing\nthe depth and width of the network whilst keeping computation operations\nconstant. The designers of this network coined the term “inception\nmodules” to describe an optimized network structure with skipped\nconnections that is used as a building block. This inception module is\nrepeated spatially by stacking with occasional max-pooling layers to reduce\ndimensionality to a manageable level for computation.\nThe Xception  [25]  CNN was developed\nby Google Inc. as an “extreme” version of the Inception model.\nThe Inception modules described above are replaced with depth wise separable\nconvolutions. This Xception was shown to outperform Inception on a\nlarge-scale image classification dataset (comprising 350 million images of\n17,000 classes).\nThe InceptionResNetV2 CNN  [26] \ncombines the Inception and Resnet architecture with the objective of\nachieving high classification performance using a ResNet architecture with\nthe low computation cost of the Inception architecture.\nThe NASNet (Large) CNN  [28]  has been\ndeveloped by Google Brain as a data driven dynamic network that uses\nreinforcement learning to determine an optimal network structure for the\nimage classification task at hand.\nThe DenseNet 121 CNN  [27]  uses\nshorter connections between layers in order to allow more accurate and\nefficient training on very deep networks.\nThe Models discussed in the earlier section was firstly tuned using Keras-tune to\ndetermine an optimum range of learning rate, hidden network size and dropout\nrate. From this process, optimal hyperparameter ranges were determined to be:\n • Learning Rate = 10 −3  –\n10 −5 , • Hidden Layer Size = 8 – 96 neurons, • Dropout =0.1 – 0.2\nLearning Rate = 10 −3  –\n10 −5 ,\nHidden Layer Size = 8 – 96 neurons,\nDropout =0.1 – 0.2\nEach model was then trained 5 times over 100 epochs with precision, recall,\ntraining/testing accuracy and loss metrics captured along with training curves\nand confusion matrices for further analysis.\nThe test was repeated for learning rates between 10 −3  and\n10 −5  with order-of-magnitude increments. The hidden layer\nsize was also varied between 8 and 96. The batch size was varied between 2 and\n16. Each classifier was trained on the ImageNet  [95]  weights for transfer learning. Finally, where models\nconverged well, the best training hyperparameters were selected by inspection of\ntraining curves. The number of training epochs was then adjusted to prevent\noverfitting. The training and testing were then repeated with selected epochs\nand optimized hyperparameters to obtain performance scores.\nThe testing results as shown in  Table\n3  that the simpler VGG classifiers were more trainable on all three\nimage modes and provided more consistent results across all three image modes.\nWe also noted that ultrasound provided best classification results across all\ndeep learning models compared to the CT and X-Ray image modes. The more complex\nmodels tended to either overfit in early epochs (<10) or failed to\nconverge at all. Where reasonable results were obtained from the more complex\nmodels training curves typically showed overfitting and somewhat erratic\ntraining behavior in several cases. We also found that the more complex model\ntrainability was highly dependent upon initial model hyperparameter choice,\nwhereas the VGG classifiers produce good results for a wider range of\nhyperparameter choices. TABLE 3 Model Performance Summary Model Image mode Training Notes Testing F1 (avg) VGG16 X-Ray Converged well. Overfitting evident\nat 30 epochs. 0.79 Ultrasound Converged well. Overfitting not\nevident at 100 epochs. 0.99 CT Converged. Overfitting evident from\n5 epochs. 0.79 VGG19 X-Ray Converged well. Overfitting not\nevident at 100 epochs. 0.87 Ultrasound Converged well. Overfitting not\nevident at 100 epochs. 0.99 CT Converged. Overfitting evident after\n20 epochs. 0.78 Xception X-Ray Converged poorly. Overfitting\nevident after 10 epochs 0.76 Ultrasound Converged. Overfitting evident after\n20 epochs. 0.82 CT Did not converge. Overfitting\nevident immediately 0.70 InceptionResNet X-Ray Converged poorly. Overfitting\nevident after 10 epochs. 0.73 Ultrasound Converged. Overfitting evident after\n5 epochs. 0.89 CT Did not converge. Overfitting\nevident after 1st epoch. 0.63 InceptionV3 X-Ray Converged poorly. Overfitting\nevident after 10 epochs. 0.75 Ultrasound Converged. Overfitting evident after\n10 epochs. 0.87 CT Did not converge. Overfitting\nevident after 2 epochs. 0.71 NASNetLarge X-Ray Did not converge. Overfitting\nevident after 10 epochs. 0.64 Ultrasound Converged. Overfitting evident after\n10 epochs. 0.80 CT Did not converge. Overfitting\nevident immediately. 0.64 DenseNet121 X-Ray Converged poorly. Overfitting\nevident after 10 epochs. 0.74 Ultrasound Converged poorly. Overfitting\nevident immediately. 0.66 CT Converged. Overfitting evident after\n8 epochs. 0.75 ResNet50V2 X-Ray Converged. Overfitting evident after\n20 epochs. 0.75 Ultrasound Converged well. Overfitting evident\nimmediately. 0.93 CT Converged poorly. Overfitting\nevident immediately. 0.66\nFinally, we noticed that the more complex models exhibited a higher training\nmetrics deviation between epochs with randomly selected train/test splits. We\nbelieve the smaller data size and high fine-grained variability within the\ndatasets were detected by the sensitive complex models, thus resulting in poorer\nperformances. We expect complex model performance to improve with larger and\nbetter-quality data.\nBased on our initial testing results, we have chosen the VGG19 model for our\nmultimodal image classification testing in this study. We anticipate that future\nnovel pandemics can also be expected to initially produce small, low quality\nmedical image datasets and suggest that our findings are likely to extend to\nsimilar future applications with such challenging datasets.\nThe three modes of data (X-Ray, CT and Ultrasound) mainly used to understand lung\nconditions for a covid-19 patient. Through popular deep learning models, we try\nto understand their individual strength and weakness to detect COVID-19 lung\ncondition for reasonable precision performance. This is vital for a doctor for\ndecision making as each has advantages and disadvantages. Moreover, when there\nis limited time, resource, and patient condition, the doctor may need to take\ndecision based on one modality. These results would help practitioners in\nselecting appropriate models for different imaging modes thus providing critical\nsupport when time and resources are stretched in a pandemic situation like the\ncurrent COVID19.\n\nAll the experiments were performed on the University of Technology Sydney\nInteractive HPC environment under an Anaconda 3 software environment.\nExperiments were programmed using the Keras APIs with a TensorFlow 2\nbackend.\nThe server used was specified as an Intel Xeon Gold 6150 2.7GHz 18 cores (16\ncores enabled) with 24.75MB L3 Cache (Max Turbo Freq. 3.7GHz, Min 3.4GHz). The\nserver had 360GB RAM (Six Channel). This server hosted a NVIDIA Quadro P5000 GPU\n(2,560 Cores, 16GB Memory).\nVGG19 model was tuned for each image mode and each individual experiment to\nachieve the best possible results for the collated datasets. Learning rates were\nvaried between 10 −3  to 10 −6  with\norder-of-magnitude increments. The batch sizes between 2 to 16 were applied. The\nhidden layer was varied between 4 and 96 nodes. Dropout rates of 0.1 and 0.2\nwere applied. These hyperparameter ranges generated an output head architecture\nas shown in  Fig. 5 .  FIGURE 5. Head architecture of the proposed models.\nHead architecture of the proposed models.\nThe master dataset was utilized for training and testing with the VGG19\nclassifier over 5 experiments as shown in  Table 4 . TABLE 4 Datasets Used for Experiments Exp ID Image Mode Experiment Dataset 1A X-Ray Normal vs (COVID-19 and\nPneumonia) (400  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$\\times$\n                                            \\end{document}  Normal) vs (190  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$\\times$\n                                            \\end{document}  Pneumonia 139  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$\\times$\n                                            \\end{document}  COVID-19) 1B Ultrasound Normal vs (COVID-19 and\nPneumonia) 226  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$\\times$\n                                            \\end{document}  Normal vs (235  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$\\times$\n                                            \\end{document}  COVID-19 220  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$\\times$\n                                            \\end{document}  Pneumonia) 2A X-Ray COVID-19 vs Pneumonia 139  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$\\times$\n                                            \\end{document}  COVID-19 vs 190  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$\\times$\n                                            \\end{document}  Pneumonia 2B Ultrasound COVID-19 vs Pneumonia 235  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$\\times$\n                                            \\end{document}  COVID-19 vs 220  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$\\times$\n                                            \\end{document}  Pneumonia 3A CT COVID-19 vs Non COVID 349  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$\\times$\n                                            \\end{document}  COVID-19 vs 397  \n \\documentclass[12pt]{minimal}\n\\usepackage{amsmath}\n\\usepackage{wasysym}\n\\usepackage{amsfonts}\n\\usepackage{amssymb}\n\\usepackage{amsbsy}\n\\usepackage{upgreek}\n\\usepackage{mathrsfs}\n\\setlength{\\oddsidemargin}{-69pt}\n\\begin{document}\n}{}$\\times$\n                                            \\end{document}  Non COVID\nProcessed dataset is available at:\n https://github.com/mhorry/N-CLAHE-MEDICAL-IMAGES\nWith the selected VGG19 model for each experiment listed in  Table 4 , we first conducted the\nextensive performance tuning by adjusting multiple parameters including learning\nrate, batch size, node size and drop rate. The effects of learning rate, batch\nsize and hidden layer size hyperparameter selection on the accuracy metric of\nexperiment 1A is shown in  Fig. 6 . We\nnoted that dropout rate had only a minimal effect on the model accuracy except\nat the highest learning rate of 10 −3  and lowest learning rate\nof 10 −6  where a dropout rate of 0.2 proved to be more stable\nthan a dropout rate of 0.1. For learning rates for 10 −3  and\n10 −4  the dropout rate selection has no discernable effect\non model accuracy.  FIGURE 6. Model sensitivity to hyperparameters for experiment 1A.\nModel sensitivity to hyperparameters for experiment 1A.\nLearning rate, batch size and hidden layer size all affected model accuracy. The\nfirst observation is that learning rates of 10 −4  and\n10 −5  provided higher model accuracy with\n10 −5  achieving more consistent results. There is a\ntendency for accuracy to improve with batch size increase at learning rates of\n10 −3  and 10 −6  but at learning rates of\n10 −4  and 10 −5  this tendency is not\napparent. Finally, there is also a trend towards higher accuracy with a larger\nhidden layer size that is most noticeable at 10 −3 . Taking a\nlearning rate of 10 −5  as achieving consistent high accuracy,\nwe can then suggest from this analysis that a hidden layer size ranging from 64\nto 96 and batch size of 4 could generally be expected to provide the most\naccurate results for this experiment. Through similar analysis for each\nexperiment in  Table 4 , we have\nidentified the best parameter settings for each experiment as shown in  Table 5 . TABLE 5 Experiment Results for Three Image Modes Image mode Experiment Parameters Classification Results X-Ray 1A LR:\n10 −5 DR: 0.1 BS:\n4 HS: 64 Epochs: 100 COVID-19 + Pneumonia P: 0.85 R: 0.83 F1:\n0.84 Normal P: 0.86 R: 0.88 F1:\n0.87 Ultrasound 2A LR:\n10 −5 DR: 0.2 BS:\n2 HS: 64 Epochs: 100 COVID-19 + Pneumonia P: 0.99 R: 0.97 F1:\n0.98 Normal P: 0.94 R: 0.98 F1:\n0.96 X-Ray 1B LR:\n10 −3 DR: 0.2 BS:\n8 HS: 8 Epochs: 100 COVID-19 P: 0.86 R: 0.86 F1:\n0.86 Pneumonia P: 0.89 R: 0.89 F1:\n0.89 Ultrasound 2B LR:\n10 −5 DR: 0.2 BS:\n2 HS: 64 Epochs: 100 COVID-19 P: 1.00 R: 1.00 F1:\n1.00 Pneumonia P: 1.00 R: 1.00 F1:\n1,00 CT 3A LR:\n10 −3 DR: 0.2 BS:\n4 HS: 16 Epochs: 70 COVID-19 P: 0.79 R: 0.83 F1:\n0.81 Non COVID P: 0.84 R: 0.81 F1:\n0.83\nThe results of the five experiments are listed in  Table 5 . For experiments classifying COVID-19 and\nPneumonia vs Normal (1A and 2A) we found that the Ultrasound mode provided the\nbest results with a sensitivity of 97% and positive predictive value of\n99% compared to X-Ray with 83% and 85% respectively. For\nexperiments classifying COVID-19 vs Pneumonia (1B and 2B) we again found that\nthe Ultrasound mode provided the best results with a sensitivity of 100%\nand a positive predictive value of 100% compared to X-Ray with\nsensitivity of 86% and positive predictive value of 86%. The CT\nimaging mode was found to have a sensitivity of 83% and positive\npredictive value of 79% in classifying COVID-19 vs non COVID-19 scans.\nAll experiments resulted in F1 scores exceeding 80% which is a good\nresult given the relatively small and variable quality data corpus\navailable.\nThe learning curves for each experiment are shown in  Fig. 7 . The training curves for both Ultrasound\nexperiments (2A and 2B) are close to ideal. The training curves for the X-Ray\nexperiments (1A and 2A) are also very good, although the curve for experiment 1B\ndoes show some signs of erratic learning patterns, which is the expected result\nof the highly variable image quality in the COVID-19 data set. The learning\ncurve for the CT mode experiment (3A) is very erratic – even though the\nmodel did train, overfitting is arguably apparent after the epoch 50. Once again\nthis is the expected result considering significant variation in the CT image\ndata sets.  FIGURE 7. Learning curves for different modes.\nLearning curves for different modes.\nThe confusion matrices in  Fig. 8  provides\nan indication of the false-negative and false-positive results of our\nexperiments. Minimization of false negative predictions is important in the\nmedical context since false reassurance may lead to diagnostic and treatment\ndelay resulting in poor medical outcomes, patient mental distress, community\nloss in confidence relating to medical services and legal consequences  [96] . False negative predictions for the\nUltrasound mode experiments were very low at 1 and 0 for experiments 2A and 2B,\nrespectively. False negative predictions for the X-Ray mode experiments were\nhigher with 11 and 4 for experiments 1A and 1B respectively. The CT mode\n(experiment 3A) also performed poorly in this respect with 12 false negatives.\nOnce again, the limited sample size and variable quality of the COVID-19 data\nsets used for the X-Ray and CT experiments are the most likely cause of the\nrelatively high number of false negatives for experiments 1A, 1B and 3A.  FIGURE 8. Confusion matrix for different modes.\nConfusion matrix for different modes.\nAs previously noted, false negatives generated by the Keras class prediction\nthreshold of 0.5 were high in the case of the CT and X-Ray imaging modes. We\nthen performed adjustments in the class prediction threshold in 5%\nincrements from 0.5 to 0.75. We successfully reduced false negatives, at the\ncost of increasing false positives. For experiment 1A we reduced false negatives\nfrom 11 to 8 using a threshold of 0.65. This raised the sensitivity of this test\nfrom 83% to 85% with a loss in positive predictive value from\n85% to 73%. For experiment 1B we reduced false negatives from 4 to\n2 using a threshold of 0.75. This raised the sensitivity of this test from\n86% to 93% whilst reducing positive predictive value from\n86% to 79%. For experiment 3A we reduced false negatives from 12\nto 7 using a threshold of 0.7. This raised the sensitivity of this test from\n83% to 90% with a reduction in positive predictive value from\n79% to 78%. The confusion matrices associated with these results\nare shown in  Fig. 9 .  FIGURE 9. Reduction of false negatives using thresholding.\nReduction of false negatives using thresholding.\nOverall, the results show that deep learning models were performing extremely\nwell with the ultrasound image mode. The X-Ray and CT modes are challenging with\nthe available COVID-19 datasets.\n\nWe have demonstrated that with current limited and challenging COVID-19 datasets,\nVGG19 model could be used to develop suitable deep learning-based tools for COVID-19\ndetection. The model is capable of classifying both Pneumonia vs Normal and COVID-19\nvs Pneumonia conditions for multiple imaging modes including X-Ray, Ultrasound, and\nCT scan.\nWith very little data curation, we achieved considerable classification results using\nVGG19 from all imaging modes. Perhaps the most interesting observation is that the\npre-trained models tuned very effectively for the Ultrasound image samples, which to\nthe untrained eye appeared noisy and difficult to interpret. Both training curves\nand confusion matrix for both Ultrasound experiments are close to ideal. VGG19 also\ntrained well against the X-Ray image corpus however, without modified thresholding\nwe found that the proportion of false negatives was concerning but not unexpected\ngiven data quality challenges. Our finding that experiment 1A/2A yielded lower F1\nscores and higher false negatives than experiments 1B/2B was unexpected since the\nmanifestation of COVID-19 is itself a form of viral pneumonia. This may indicate\nthat despite our attempts to remove sampling bias using N-CLAHE pre-processing there\nmay still be systematic differences in the COVID-19 image data sets that leads the\nVGG19 classifier to more easily distinguish the COVID-19 images from the pneumonia\nimages. A future research direction could be to isolate the lung field by\nsegmentation for all image samples in order to remove noise and further reduce\nsampling bias. Our lower results against the CT image corpus were not surprising\nsince the CT image slices available were not from a uniform patient location and\ndisplayed extremely high variability in both form and content.\nOur study uncovers the challenging characteristics of the limited COVID-19 image\ndatasets. This should be helpful for practitioners aiming to use these datasets for\ntheir research and development. We provided a pre-processing pipeline aimed to\nremove the sampling bias and improve image quality. Our preprocessed images are also\nmade openly available for others to use. During our initial model selection\nexperiment, we also found that both VGG16 and VGG19 classifiers provided good\nresults within the experimental constraints of the small number of currently\navailable COVID-19 medical images. While deeper networks generally struggled, they\nwill perform better when larger datasets are available which will reduce the impact\nof data quality variation.\nIt is inevitable that the initial publicly available medical images for novel medical\nconditions such as COVID-19 will be low in number and poor in quality. In this\nsituation we conclude that the VGG19 classifier with transfer learning provides a\nfast and simple to implement machine learning model for multiple imaging modes\nproviding good results that may lead to clinically useful diagnostic tools.\nDespite our promising results, we would urge great caution in the development of\nclinical diagnostic models using currently available COVID-19 image dataset. The\neffect of a false positive diagnosis of COVID-19 on an individual is the isolation\nof the individual and their contract traces and the mental anguish and stress caused\nby both the prognosis and the social isolation. A false positive COVID-19 diagnosis\ncould result in an inappropriate course of treatment. The effects of a false\nnegative COVID-19 diagnosis would also be devastating for the individual if that\ndiagnosis led to an inappropriate lack of treatment, and also for the community\nsince cautions against COVID-19 transmission may not be appropriately applied\nresulting in the further community spread of the disease.\nAs a higher quality corpus of COVID-19 diagnostic image data becomes available, it\nmay be possible to produce clinically trusted deep learning-based models for the\nfast diagnosis of COVID-19 as distinguished from similar conditions such as\npneumonia. Such a tool would prove invaluable in practice, where other diagnostic\ntests for COVID-19 are either unavailable or unreliable. As the COVID-19 spread\nprogresses throughout remote and economically challenged locations, an ability to\ndiagnose COVID-19 from a readily available and portable medical imaging equipment\nsuch as X-Ray and Ultrasound machines would help slow the spread of the disease and\nresult in a better medical outcome for the population.\nData fusion concept allows us to combine multiple modes of data to improve model\nclassification performance. Although data fusion comes with its own set of\nchallenges  [97] ,  [98] , it has been used successfully in other application\nareas such as remote sensing  [99] – [101] , action detection  [102] , and medical diagnosis and imaging  [103] ,  [104] . We\nplan to extend our study with multimodal data fusion when sufficient data is\navailable.","source_license":"CC-BY-4.0","license_restricted":false}