Novel applications of Convolutional Neural Networks in the age of Transformers

Novel applications of Convolutional Neural Networks in the age of Transformers | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Novel applications of Convolutional Neural Networks in the age of Transformers Tansel Ersavas, Martin A. Smith, John S. Mattick This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3868861/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 30 Apr, 2024 Read the published version in Scientific Reports → Version 1 posted 9 You are reading this latest preprint version Abstract Convolutional Neural Networks (CNNs) have been central to the Deep Learning revolution and played a key role in initiating the new age of Artificial Intelligence. However, in recent years newer architectures such as Transformers have dominated both research and practical applications. While CNNs still play critical roles in many of the newer developments such as Generative AI, they are far from being thoroughly understood and utilised to their full potential. Here we show that CNNs can recognise patterns in images with scattered pixels and can be used to analyse complex datasets by transforming them into pseudo images in a standardised way for any high dimensional dataset, representing a major advance in the application of CNNs to datasets such as in molecular biology, text, and speech. We introduce a simple approach called DeepMapping , which allows analysis of very high dimensional datasets without intermediate filtering and dimension reduction, thus preserving the full texture of the data, enabling the ability to detect small perturbations. We also demonstrate that DeepMapper is superior in speed and on par in accuracy to prior work in processing large datasets with large numbers of features. Physical sciences/Mathematics and computing/Computational science Biological sciences/Computational biology and bioinformatics/Machine learning Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Introduction There are exponential increases in data 1 especially from highly complex systems, whose non-linear interactions and relationships are not well understood, and which can display major or unexpected changes in response to small perturbations, known as the ‘Butterfly effect’ 2 . In domains characterised by high-dimensional data, traditional statistical methods and Machine Learning (ML) techniques make heavy use of feature engineering that incorporates extensive filtering, selection of highly variable parameters, and dimension reduction techniques such as Principal Component Analysis (PCA) 3 . Most current tools filter out smaller changes in data, mostly considered artefacts or `noise`, which may contain information that is paramount to understanding the nature and behaviour of such highly complex systems 4 . The emergence of Deep Learning (DL) offers a paradigm shift. DL algorithms, underpinned by adaptive learning mechanisms, can discern both linear and non-linear data intricacies, and open avenues to analyse data that is not possible or practical by conventional techniques 5 , particularly in complex domains such as image, temporal sequence analysis, molecular biology, and astronomy 6 . DL models, such as Convolutional Neural Networks (CNNs) 7 , Recurrent Neural Networks (RNNs) 8 , Generative Network s 9 and Transformers 10 , have demonstrated exceptional performance in various domains, such as image and speech recognition, natural language processing, and game playing 6 . CNNs and LSTMs were found to be great tools to predict behaviour of so called `chaotic` systems 11 . Modern DL systems often surpass human-level performance, and challenge humans even in creative endeavours. CNNs utilise a unique architecture that comprises several layers, including convolutional layers, pooling layers, and fully connected layers, to process and transform the input data hierarchically 5 . CNNs have no knowledge of sequence, and therefore are generally not used in analysing time-series or similar data, which is traditionally attempted with Recurrent Neural Networks (RNNs) 12 and Long Short-Term Memory networks (LSTMs) 8 due to their ability to capture temporal patterns. Where CNNs have been employed for sequence or time-series analysis, 1-dimensional (1D) CNNs have been selected because of their vector based 1D input structure 13 . However, attempts to analyse such data in 1D CNNs do not always give superior results 14 . In addition, GPU (Graphical Processing Units) systems are not always optimised for processing 1D CNNs, therefore even though 1D CNNs have fewer parameters than 2-dimensional (2D) CNNs, 2D CNNs can outperform 1D CNNs 15 . Transformers , introduced by Vaswani and others 10 , have recently come to prominence, particularly for tasks where data is in the form of time series or sequences, in domains ranging from language modelling to stock market prediction 16 . Transformers leverage self-attention, a key component that allows a model to weigh and focus on various parts of an input sequence when producing an output, enabling the capture of long-range dependencies in data. Unlike CNNs, which use local receptive fields, self-attention weighs the significance of various parts of the input data 17 . Following success with sequence-based tasks, Transformers are being extended to image processing. Vision-Transformers in object detection 18 , Detection Transformers 19 and lately Real-time Detection Transformers all claim superiority over CNNs 20 . However, their inference operations demand far more resources than CNNs and trail CNNs in flexibility. They also suffer similar augmentation problems as CNNs. More recently, Retentive-Networks have been offered as an alternative to Transformers 21 and may soon challenge the Transformer architecture. CNNs can recognise dispersed patterns Even though CNNs are widely used, there are some misconceptions, notably that CNNs are largely limited to image data, and require established spatial relationships between pixels in images, both of which are open to challenge. The latter is of particular importance when considering the potential of CNNs to analyse complex non-image datasets, whose data structures are arbitrary. Moreover, while CNNs are universal function approximators 22 , they may not always generalise 23 , especially if they are trained on data that is insufficient to cover the solution space 24 . It is also known that they can spontaneously generalise even when supplied with a small number of samples during training after overfitting, called `grokking` 25 , 26 . CNNs can generalise from scattered data if given enough samples, or if they grok, and this can be determined by observing changes to training versus testing accuracy and loss. Non-image processing with CNNs While CNNs have achieved remarkable success in computer vision applications, such as image classification and object detection 7 , 27 , they have also been employed in other domains to a lesser degree with impressive results, including: (i) natural language processing, text classification, sentiment analysis and named entity recognition, by treating text data as a one-dimensional image with characters represented as pixels 16 , 28 ; (ii) audio processing, such as speech recognition, speaker identification and audio event detection, by applying convolutions over time frequency representations of audio signals 29 ; (iii) time series analysis, such as financial market prediction, human activity recognition and medical signal analysis, using one-dimensional convolutions to capture local temporal patterns and learn features from time series data 30 ; and (iv) biopolymer (e.g., DNA) sequencing, using 2D CNNs to accurately classify molecular barcodes in raw signals from Oxford Nanopore sequencers using a transformation to turn a 1D signal into 2D images - improving barcode identification recovery from 38% to over 85% 31 . Indeed, CNNs are not perfect tools for image processing as they do not develop semantic understanding of images even though they can be trained to do semantic segmentation 32 . They cannot easily recognise negative images when trained with positive images 33 . CNNs are also sensitive to the orientation and scale of objects and must rely on augmentation of image datasets, often involving hundreds of variations of the same image 34 . There are no such changes in the perspective and orientation of data converted into flat 2D images. In the realm of complex domains that generate huge amounts of data, augmentation is usually not required for non-image datasets, as the datasets will be rich enough. Moreover, introducing arbitrary augmentation does not always improve accuracy, indeed, introducing hand-tailored augmentation may hinder analysis 35 . If augmentation is required, it can be introduced in a data-oriented form, but even when using automated augmentation such as AutoAugment 35 or FasterAutoAugment 36 , many of the augmentations (such as shearing, translation, rotation, inversion, etc.) should not be used, and the result should be tested carefully, as augmentation may introduce artefacts. A frequent problem with handling non-image datasets with many variables is noise. Many algorithms have been developed for noise elimination, most of which are domain specific. CNNs can be trained to use the whole input space with minimal filtering and no dimension reduction, and can find useful information in what might be ascribed as ‘noise’ 4 , 37 . Indeed, a key reason to retain ‘noise’ is to allow discovery of small perturbations that cannot be detected by other methods 11 . Conversion of non-image data to artificial images for CNN processing Transforming sequence data to images without resorting to dimension reduction or filtering offers a potent toolset for discerning complex patterns in time series and sequence data, which potentiates the two major advantages of CNNs compared to RNNs, LSTMs and Transformers . First, CNNs do not depend on past data to recognise current patterns, which increases sensitivity to detect patterns that appear in the beginning of time-series or sequence data. Second, 2D CNNs are better optimised for GPUs and highly parallelizable, and are consequently faster than other current architectures, which accelerates training and inference, while reducing resource and energy consumption during in all phases including image transformation, training, and inference significantly. Image data such as MNIST represented in a matrix can be classified by basic deep networks such as Multi-level Perceptrons (MLP) by turning their matrix representation to vectors (Fig. 1a). Using this approach analysis of images becomes increasingly complex as the image size grows, increasing the input parameters of MLP and the computational cost exponentially. On the other hand, 2D CNNs can handle the original matrix much faster than MLP with equal or better accuracy and scale to much larger images. Just like how a simple neural network analyses a 2D image by turning it into a vector, the reciprocal is also true - data in a vector can be converted to a 2D matrix (Fig. 1b). Vectors converted to such matrices form arbitrary patterns that are incomprehensible to human eye. A similar technique for such mapping has also been proposed by Kovelarchuk et al using another algorithm called CPC-R 38 . Attribution An important aspect of any analysis is to be able to identify those variables that are most important and the degree to which they contribute to a given classification. Identifying these variables is particularly challenging in CNNs due to their complex hierarchical architecture, and many non-linear transformations 39 . To address this problem many ‘attribution methods’ have been developed to try to quantify the contribution of each variable (e.g., pixels in images) to the final output for deep neural networks and CNNs 40 . Saliency maps serve as an intuitive attribution and visualisation tool for CNNs, spotlighting regions in input data that significantly influence the model's predictions 27 . By offering a heatmap representation, these maps illuminate key features that the model deems crucial, thus aiding in demystifying the model's decision-making process. For instance, when analysing an image of a cat, the saliency map would emphasise the cat's distinct features over the background. While their simplicity facilitates understanding even for those less acquainted with deep learning, saliency maps do face challenges, particularly their sensitivity to noise and occasional misalignment with human intuition 41-43 . Nonetheless, they remain a pivotal tool in enhancing model transparency and bridging the interpretability gap between ML models and human comprehension. Several methods have been proposed for attribution, including Guided Backpropagation 44 , Layer-wise Relevance Propagation 45 , Gradient-weighted Class Activation Mapping 46 , Integrated Gradients 47 , DeepLIFT 48 , and SHAP (SHapley Additive exPlanations) 49 . Many of these methods were developed because it is challenging to identify important input features when there are different images with the same label (e.g., ‘bird’ with many species) presented at different scales, colours, and perspectives. In contrast, most non-image data does not have such variations, as each pixel corresponds to the same feature. For this reason, choosing attributions with minimal processing is sufficient to identify the salient input variables that have the maximal impact on classification. DeepMapper Here we introduce a new analytical pipeline, DeepMapper , which applies a non-indexed or indexed mapping to the data representing each data point with one pixel, enabling the classification or clustering of data using 2D CNNs. This simple direct mapping has been tried by others but has not been tested with datasets with sufficiently large amounts of data in various conditions. The pipeline includes conversion of data, separation to training and validation, assessment of training quality, attribution, and accumulation of results in a pipeline. The pipeline is run multiple times until a consensus is reached. The significant variables can then be identified using attribution and exported appropriately. The DeepMapper architecture is shown in Fig. 2. The complete algorithm of DeepMapper is detailed in the methods section and the Python source code is supplied at GitHub 50 . Results We present a simple example to demonstrate that CNNs can readily interpret data with a well dispersed pattern of pixels, using the MNIST dataset, which is widely used for hand-written image recognition and which humans as well as CNNs can easily recognise and classify based on the obvious spatial relationships between pixels (Fig. 3 ). This dataset is a more complicated problem than datasets such as the Gisette dataset 51 that was developed to distinguish between 4 and 9. It includes all digits and uses a full randomisation of pixels, and can be regenerated with the script supplied 50 and changing the seed will generate different patterns. We randomly shuffled the data in Fig. 3 using the same seed 50 to obtain 60,000 training images such as those shown on the right side of each digit, and validated the results with a separate batch of 20,000 images (Fig. 3 ). Although the resulting images are no longer recognizable by eye, a CNN has no difficulty distinguishing and classifying each pattern with ~ 2% testing error compared to the reference data (Fig. 4 ). This result demonstrates that CNNs can accurately recognise global patterns in images without reliance on local relationships between neighbouring pixels. It also confirms the finding that shuffling images only marginally increases training loss 23 and extends it to testing loss (Fig. 4 ). Testing DeepMapper Finding slight changes in very few variables in otherwise seemingly random datasets with large numbers of variables is like finding a needle in a haystack. Such differences in data are almost impossible to detect using traditional analysis tools because small variations are usually filtered out before analysis. We devised a simple test case to determine if DeepMapper can detect one or more variables with small but distinct variations in otherwise randomly generated data. We generated two sets of 5,000 data items with 18,225 numeric variables. Using PyTorch’s uniform random algorithms 52 that set 17,998 of these to random numbers in the range of 0–1, and two of the variables into two distinct groups as seen in Table 1. We call this type of dataset ‘Needle in a haystack’ (NIHS) dataset, where very small amounts of data with small variance is hidden among a set of random variables that is order(s) of magnitude greater than the meaningful components. We provide a script that can generate this and similar datasets among the source supplied 50 . DeepMapper was able to accurately classify the two datasets (Fig. 4 ). Furthermore, using attribution DeepMapper was also able to determine the two datapoints that have small but different variances in the two classes. Note that DeepMapper may not always find all the changes as neural network initialisation of weights is a stochastic process. However, DeepMapper o vercomes this matter via multiple iterations to establish acceptable training and testing accuracies as described in the Methods. Comparison of DeepMapper with DeepInsight DeepInsight is the most general approach published to date for converting non-image data into image-like structures, with the claim that these processed structures allow CNNs to capture complex patterns and features in the data 53 . DeepInsight offers an algorithm to create images that have similar features collated into a “well organised image form”, or by applying one of several dimensionality reduction algorithms (e.g., t-SNE, PCA or KPCA) 53 . However, these algorithms add computational complexity, potentially eliminate valuable information, limit the abilities of CNNs to find small perturbations, and make it more difficult to use attribution to determine most notable features impacting analysis as multiple features may overlap in the transformed image. In contrast DeepMapper uses a direct mapping mechanism where each feature corresponds to one pixel. To identify important input variables, DeepInsight incorporates DeepFeature 54 using an elaborate mechanism to associate image areas identified by attribution methods to the input variables. DeepMapper uses a simpler approach as each pixel corresponds to only one variable and can use any of the attribution methods to link results to its input space. While both DeepMapper and DeepInsight follow the general idea that non-image data can be processed with 2D CNNs, DeepMapper uses a much simpler and faster algorithm, while DeepInsight chooses a sophisticated set of algorithms to convert non-image data to images, dramatically increasing computational cost. The DeepInsight conversion process is not designed to utilise GPUs so cannot be accelerated by better hardware, and the obtained images may be larger than the number of data points, also impacting performance. The DeepInsight manuscript offers various examples of data to demonstrate its abilities. However, many of the examples use low dimensions (20 to ~ 4,000 features) while today’s complex datasets may regularly require tens of thousands to millions of features such as in genome analysis in biology and radio-telescope analysis in astronomy. As such, several examples provided by DeepInsight have insufficient dimensions for a sophisticated mechanism such as DeepMapper , which should ideally have 10,000 or more dimensions as required by modern complex datasets. DeepInsight examples include a speech dataset from the TIMIT corpus with 39 dimensions, Relathe (text) dataset, which is derived from newsgroup documents and partitioned evenly across different newsgroups. It contains 1,427 samples and 4,322 dimensions. The ringnorm-DELVE , which is an implementation of Leo Breiman’s ringnorm example, is a 20 dimensional, 2 class classification with 7,400 samples 53 . Another example, Madelon , introduced an artificially generated dataset 2,600 samples and 500 dimensions, where only 5 principal and 20 derived variables containing information. Instead, we used a much more complicated example than Madelon , an NIHS dataset 50 that we used to test DeepMapper in the first place. We attempted to run DeepInsight with NIHS data, but we could not get it to train properly and for this reason we cannot supply a comparison. The most complex problem analysed by DeepInsight was a public RNA sequencing gene expression dataset from TCGA ( https://cancergenome.nih.gov/ ) containing 6,216 samples of 60,483 genes or dimensions. DeepMapper exhibited much more improved processing speed with near identical accuracy (Table 2, Fig. 5 ). Discussion CNNs are fundamentally sophisticated pattern matchers that can establish intricate mappings between input features and output representations 6 . They excel at transforming various inputs into outputs, including identifying classes or bounding boxes, through a series of operations involving convolution, pooling, and activation functions 7 , 55 . Even though CNNs are in the centre of many of today’s revolutionary AI systems from self-driving cars to generative AI systems such as Dall-E-2 , MidJourney and Stable Diffusion , they are still not well understood nor efficiently utilised, and their usage beyond image analysis has been limited. While CNNs used in image analysis are constrained historically and practically to a 224 x 224 matrix or a similar fixed size input, this limitation arises for pre-trained models. When CNNs have not been pre-trained, one can select a much wider variety of sizes as input shape depending on the CNN architecture. Some CNNs are more flexible in their input size that implemented with adaptive pooling layers such as ResNet18 using adaptive pooling 56 . This provides flexibility to choose optimal sizes for the task in hand for non-image applications, as most non-image applications will not use pre-trained CNNs. Here we have demonstrated uses of CNNs that are outside the norm. There is a need for analysis of complex data with many thousands of features that are not primarily images. There is also a lack of tools that offer minimal conversion of non-image data to image-like formats that then can easily be processed with CNNs in classification and clustering tasks. As a lot of this data is coming from complex systems that have a lot of features, DeepMapper offers a way of investigating such data in ways that may not be possible with traditional approaches. Even though DeepMapper currently uses CNN as its AI component, alternative analytic strategies can easily be substituted in lieu of CNN with minimal changes, such as Vision Transformers 18 or RetNets 21 , which have great potential for this application. While Transformers and RetNets have input size limitations for inference in terms of number of tokens. Vision Transformers can handle much larger inputs by dividing images to segments that incorporate multiple pixels 18 . This type of approach would be applicable to both Transformers and RetNets , and future architectures. DeepMapping can leverage these newer architectures, and others, in the future 56 . Methods DeepMapper is developed to implement an approach to process high-dimensional data without resorting to excessive filtering and dimension reduction techniques that eliminate smaller perturbations in data to be able to identify those differences that would otherwise be filtered out. The following algorithm is used to achieve this result: Read and setup the running parameters. Read the data into a tabulated form in the form of observations, features, and outcome (in the form of labels, or if self-supervised, the input itself). Identify features and labels. Do only basic filtering that eliminates observations or features if all of them are 0 or empty. Normalise features. Transform tabulated data to tensors of rank 2. If the analysis is supervised, then transform labels to output tensors. Begin iteration: Separate the data into training and testing, and validation groups. Train on the dataset for required number of epochs, until reaching satisfactory testing accuracy and loss, or maximum a pre-determined number of iterations. If satisfactory testing results are obtained, then: Perform attributions. Accumulate attribution results. If training is satisfactory: Tabulate attribution results. Save the model. Report results. The results of DeepMapper analysis can be used in 2 ways: Supervised: DeepMapper produces a list of features that played a prominent role in the differentiation of classes Self-supervised: Highlight features that are the most important features in differentiating observations from each other in a non-linear fashion. The output can be used as an alternative feature selection tool for dimension reduction In both modes, any hidden layer can be examined as latent space. A special bottleneck layer can be introduced to reduce dimensions for clustering purposes. Declarations Acknowledgements We thank Murat Karaorman, Mitchell Cummins, and Fatemeh Vafaee for helpful advice and comments on the manuscript. This research is supported by an Australian Government Research Training Program Scholarships RSAI8000 and RSAP1000 to T.E., a Fonds de Recherche du Quebec Santé Junior 1 Award 284217 to M.A.S., and UNSW SHARP grant RG193211 to J.S.M. Author contributions T.E. developed the methods, implemented DeepMapper and produced the first draft of the paper. J.S.M. provided advice, structured the paper, and edited it for improved readability and clarity. M.A.S. provided advice and edited the paper. Competing interests Authors claim no competing interests. References Taylor, P. Volume of data/information created, captured, copied, and consumed worldwide from 2010 to 2020, with forecasts from 2021 to 2025. https://www.statista.com/statistics/871513/worldwide-data-created/ (2023). Ghys, É. The butterfly effect. in The Proceedings of the 12th International Congress on Mathematical Education: Intellectual and attitudinal challenges , pp. 19–39 (Springer). https://doi.org/20.500.12657/28000/1/1001997 (2015). Jolliffe, I. T. Mathematical and statistical properties of sample principal components. Principal Component Analysis , pp. 29–61 (Springer). https://doi.org/10.1007/0-387-22440-8_3 (2002). Landauer, R. The noise is the signal. Nature 392, 658–659. https://doi.org/10.1038/33551 (1998). Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning (MIT Press). http://www.deeplearningbook.org (2016). LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444. https://doi.org/10.1038/nature14539 (2015). Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Communications of the ACM 60, 84–90. https://dl.acm.org/doi/10.1145/3065386 (2017). Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Computation 9, 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735 (1997). Goodfellow, I. et al. Generative adversarial nets. Communications of the ACM 63, 139–144. https://dl.acm.org/doi/10.1145/3422622 (2020). Vaswani, A. et al. Attention is all you need. NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems , pp. 6000–6010. https://dl.acm.org/doi/10.5555/3295222.3295349 (2017). Barrio, R. et al. Deep learning for chaos detection. Chaos 33, article 073146. https://doi.org/10.1063/5.0143876 (2023). Levin, E. A recurrent neural network: limitations and training. Neural Networks 3, 641–650. https://doi.org/10.1016/0893-6080(90)90054-O (1990). LeCun, Y. & Bengio, Y. Convolutional networks for images, speech, and time series. in The handbook of brain theory and neural networks , pp. 255–258. https://dl.acm.org/doi/10.5555/303568.303704 (MIT Press) 1998). Wu, Y., Yang, F., Liu, Y., Zha, X. & Yuan, S. A comparison of 1-D and 2-D deep convolutional neural networks in ECG classification. arXiv preprint arXiv:1810.07088 . https://doi.org/10.48550/arXiv.1810.07088 (2018). Hu, J. et al. A multichannel 2D convolutional neural network model for task-evoked fMRI data classification. Computational Intelligence and Neuroscience 2019, article 5065214. https://doi.org/10.1155/2019/5065214 (2019). Zhang, S. et al. A deep learning framework for modeling structural features of RNA-binding protein targets. Nucleic Acids Research 44, e32. https://doi.org/10.1093/nar/gkv1025 (2016). Maurício, J., Domingues, I. & Bernardino, J. Comparing vision transformers and convolutional neural networks for image classification: A literature review. Applied Sciences 13, article 5521. https://doi.org/10.3390/app13095521 (2023). Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv :2010. 11929 . https://doi.org/10.48550/arXiv.2010.11929 (2020). Carion, N. et al. End-to-end object detection with transformers. Computer Vision-ECCV 2020 (Springer), pp. 213–229. https://link.springer.com/chapter/10.1007/ 978-3-030-58452-8_13 (2020). Lv, W. et al. DETRs beat YOLOs on real-time object detection. arXiv preprint arXiv:2304.08069. https://doi.org/10.48550/arXiv.2304.08069 (2023). Sun, Y. et al. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621 . https://doi.org/10.48550/arXiv.2307.08621 (2023). Zhou, D.-X. Universality of deep convolutional neural networks. Applied and Computational Harmonic Analysis 48, 787–794. https://doi.org/10.1016/j.acha.2019.06.004 (2020). Chiyuan, Z., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM 64, 107–115. https://doi.org/10.1145/3446776 (2021). Ma, W., Papadakis, M., Tsakmalis, A., Cordy, M. & Traon, Y. L. Test selection for deep learning systems. ACM Transactions on Software Engineering and Methodology 30, article 13. https://doi.org/10.1145/3417330 (2021). Liu, Z., Michaud, E. J. & Tegmark, M. Omnigrok: grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117. https://doi.org/10.48550/arXiv.2210.01117 (2022). Power, A., Burda, Y., Edwards, H., Babuschkin, I. & Misra, V. Grokking: generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177. https://doi.org/10.48550/arXiv.2201.02177 (2022). Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 . https://doi.org/10.48550/arXiv.1312.6034 (2013). Kim, Y. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 . https://doi.org/10.48550/arXiv.1408.5882 (2014). Abdel-Hamid, O. et al. Convolutional neural networks for speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing 22, 1533–1545. https://doi.org/10.1109/TASLP.2014.2339736 (2014). Hatami, N., Gavet, Y. & Debayle, J. Classification of time-series images using deep convolutional neural networks. Proceedings Tenth International Conference on Machine Vision (ICMV 2017) 10696, 106960Y. https://doi.org/10.1117/12.2309486 (2018). Smith, M. A. et al. Molecular barcoding of native RNAs using nanopore sequencing and deep learning. Genome Research 30, 1345–1353. https://doi.org/10.1101/gr.260836.120 (2020). Emek Soylu, B. et al. Deep-Learning-based approaches for semantic segmentation of natural scene images: a review. Electronics 12, article 2730. https://doi.org/10.3390/electronics12122730 (2023). Hosseini, H., Xiao, B., Jaiswal, M. & Poovendran, R. On the limitation of Convolutional Neural Networks in recognizing negative images. 16th IEEE International Conference on Machine Learning and Applications , pp. 352–358. https://ieeexplore.ieee.org/document/8260656 (2017). Montserrat, D. M., Lin, Q., Allebach, J. & Delp, E. J. Training object detection and recognition CNN models using data augmentation. Electronic Imaging 2017, 27–36. https://doi.org/10.2352/ISSN.2470-1173.2017.10.IMAWM-163 (2017). Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V. & Le, Q. V. Autoaugment: learning augmentation policies from data. arXiv preprint arXiv:1805.09501 . https://doi.org/10.48550/arXiv.1805.09501 (2018). Hataya, R., Zdenek, J., Yoshizoe, K. & Nakayama, H. Faster AutoAugment: Learning augmentation strategies using backpropagation, in Computer Vision–ECCV 2020: 16th European Conference, Proceedings, Part XXV , pp. 1–16 (Springer). https://link.springer.com/chapter/ 10.1007/978-3-030-58595-2_1 (2020). Xiao, K., Engstrom, L., Ilyas, A. & Madry, A. Noise or signal: the role of image backgrounds in object recognition. arXiv preprint arXiv :2006. 09994 . https://doi.org/10.48550/arXiv.2006.09994 (2020). Kovalerchuk, B., Kalla, D. C. & Agarwal, B., Deep learning image recognition for non-images, in Integrating artificial intelligence and visualization for visual knowledge discovery (eds. Kovalerchuk, B., et al. ) pp. 63–100 (Springer). https://doi.org/10.1007/978-3-030-93119-3_3 (2022). Samek, W., Binder, A., Montavon, G., Lapuschkin, S. & Muller, K. R. Evaluating the visualization of what a deep neural network has learned. IEEE Transactions in Neural Networks and Learning Systems 28, 2660–2673. https://doi.org/10.1109/tnnls.2016.2599820 (2017). Montavon, G., Samek, W. & Müller, K.-R. Methods for interpreting and understanding deep neural networks. Digital Signal Processing 73, 1–15. https://doi.org/10.1016/j.dsp.2017.10.011 (2018). De Cesarei, A., Cavicchi, S., Cristadoro, G. & Lippi, M. Do humans and deep convolutional neural networks use visual information similarly for the categorization of natural scenes? Cognitive Science 45, e13009. https://doi.org/10.1111/cogs.13009 (2021). Kindermans, P.-J. et al. The (un) reliability of saliency methods, in Explainable AI: Interpreting, explaining and visualizing deep learning . Lecture Notes in Computer Science 11700, pp. 267–280 (Springer). https://doi.org/10.1007/978-3-030-28954-6_14 (2019). Zeiler, M. D. & Fergus, R. Visualizing and understanding convolutional networks. Computer Vision -- ECCV 2014 , pp. 818–833 (Fleet, D., Pajdla T., Schiele, B., & Tuytelaars, T., eds) (Springer). https://doi.org/10.1007/978-3-319-10590-1_53 (2014). Springenberg, J. T., Dosovitskiy, A., Brox, T. & Riedmiller, M. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 . https://doi.org/10.48550/arXiv.1412.6806 (2014). Binder, A., Montavon, G., Lapuschkin, S., Müller, K.-R. & Samek, W. Layer-wise relevance propagation for neural networks with local renormalization layers, in Artificial Neural Networks and Machine Learning–ICANN 2016: Proceedings 25th International Conference on Artificial Neural Networks , pp. 63–71 (Springer). https://doi.org/10.1007/978-3-319-44781-0_8 (2016). Selvaraju, R. R. et al. Grad-cam: visual explanations from deep networks via gradient-based localization. Proceedings of the 2017 IEEE international conference on computer vision , pp. 618–626. https://ieeexplore.ieee.org/document/8237336 (2017). Sundararajan, M., Taly, A. & Yan, Q. Axiomatic attribution for deep networks. Proceedings of the 34th International Conference on Machine Learning 70, 3319–3328. https://dl.acm.org/doi/10.5555/3305890.3306024 (2017). Shrikumar, A., Greenside, P. & Kundaje, A. Learning important features through propagating activation differences. Proceedings of the 34th International Conference on Machine Learning 70, 3145–3153. https://dl.acm.org/doi/10.5555/3305890.3306006 (2017). Lundberg, S. M. & Lee, S.-I. A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Machine Learning , pp. 4768–4777. https://dl.acm.org/doi/10.5555/3295222.3295230 (2017). Ersavas, T. Deepmapper. https://github.com/tansel/deepmapper (2023). Guyon, I. G. S. B.-H. A. & Dror, G. Gisette. UCI Machine Learning Repository . https://archive.ics.uci.edu/dataset/170/gisette (2008). PyTorch, torch.rand. https://pytorch.org/docs/stable/generated/torch.rand.html (2023). Sharma, A., Vans, E., Shigemizu, D., Boroevich, K. A. & Tsunoda, T. DeepInsight: A methodology to transform a non-image data to an image for convolution neural network architecture. Scientific Reports 9, article 11399. https://doi.org/10.1038/s41598-019-47765-6 (2019). Sharma, A., Lysenko, A., Boroevich, K. A., Vans, E. & Tsunoda, T. DeepFeature: feature selection in nonimage data using convolutional neural network. Briefings in Bioinformatics 22, bbab297. https://doi.org/10.1093/bib/bbab297 (2021). Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 . https://doi.org/10.48550/arXiv.1409.1556 (2014). Pytorch2, AdaptiveAvgPool2d. https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html (2023). Kokhlikyan, N. et al. Captum: a unified and generic model interpretability library for PyTorch. arXiv preprint arXiv :2009. 07896 . https://doi.org/10.48550/arXiv.2009.07896 (2020). Tables Table 1 and 2 are available in the Supplementary Files section. Additional Declarations No competing interests reported. Supplementary Files Table1and2.docx Cite Share Download PDF Status: Published Journal Publication published 30 Apr, 2024 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 12 Mar, 2024 Reviews received at journal 02 Mar, 2024 Reviewers agreed at journal 02 Mar, 2024 Reviewers agreed at journal 23 Feb, 2024 Reviewers invited by journal 27 Jan, 2024 Editor assigned by journal 27 Jan, 2024 Editor invited by journal 18 Jan, 2024 Submission checks completed at journal 18 Jan, 2024 First submitted to journal 16 Jan, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3868861","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":267767548,"identity":"3385160d-f190-40c0-b136-7b10ceb34e1e","order_by":0,"name":"Tansel Ersavas","email":"","orcid":"","institution":"UNSW Sydney","correspondingAuthor":false,"prefix":"","firstName":"Tansel","middleName":"","lastName":"Ersavas","suffix":""},{"id":267767549,"identity":"5ea904d0-6979-4ab8-bb59-fb179657d45f","order_by":1,"name":"Martin A. Smith","email":"","orcid":"","institution":"UNSW Sydney","correspondingAuthor":false,"prefix":"","firstName":"Martin","middleName":"A.","lastName":"Smith","suffix":""},{"id":267767550,"identity":"92e9e9fd-0e14-4b47-b740-50eb70f8bc7b","order_by":2,"name":"John S. Mattick","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA50lEQVRIiWNgGAWjYBACxgYwJcEgwd4AFTpAjJYDCUAtPDClhLRAFCUA7ZFIIFILcwN34uePPyzkJWe+MXv4s41Bju9GAuNnHrwO490sAXSY4WzpHHNj3jYGY8kbCczSBLRsAGlJkJPOMZNm3MaQuOFGAgMhLZt/gLVInjGT/LmNoR6ohfk3AS3bwLZIS/CYSfBuY0gwuJHAht+WZt5tFmfSJAxn9qSVSfP+AzLOPGyznINHi2F77+YbFTZ18hLHD2+T/HHGRp7vePLhG2/waWlG5UswwFMELiCPV3YUjIJRMApGAQgAADjFSH2gNXUrAAAAAElFTkSuQmCC","orcid":"","institution":"UNSW Sydney","correspondingAuthor":true,"prefix":"","firstName":"John","middleName":"S.","lastName":"Mattick","suffix":""}],"badges":[],"createdAt":"2024-01-16 05:29:11","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3868861/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3868861/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-024-60709-z","type":"published","date":"2024-05-01T00:53:58+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":49896144,"identity":"28d3af7d-3eb9-4822-be5d-7d61ba379c4c","added_by":"auto","created_at":"2024-01-19 21:52:32","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":540530,"visible":true,"origin":"","legend":"\u003cp\u003eConversion of images to vectors and vice versa.\u003cstrong\u003e a. \u003c/strong\u003eBasic operation of transformation of an image to a vector, forming a sequence representation of the numeric values of pixels. \u003cstrong\u003eb.\u003c/strong\u003e transforming a vector to a matrix, forming an image by encoding numerical values as pixels.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-3868861/v1/aeac798e6165ead65b37d1f7.png"},{"id":49895710,"identity":"060104f7-b31f-4f9c-9357-43fed6628889","added_by":"auto","created_at":"2024-01-19 21:44:33","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":206584,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eDeepMapper\u003c/em\u003e architecture. \u003cem\u003eDeepMapper\u003c/em\u003e uses sequence or multi-variate data as input. The first step of \u003cem\u003eDeepMapper\u003c/em\u003e is to merge and if required index input files to prepare them into the tensor format. The data is normalised by using log normalisation, then is folded to a matrix. Folding is performed either directly with the natural order of the data or by using the index that is generated or supplied during the data import. After folding, data is kept in temporary storage and separated to ‘train’ and ‘test’ using SciPy train test split. Training is done using either using CNNs that are supplied by the \u003cem\u003ePyTorch\u003c/em\u003e libraries, or a custom CNN supplied (\u003cem\u003eResNet18\u003c/em\u003e is used by default). Intermediary results are run through attribution algorithms supplied by the \u003cem\u003eCaptum\u003c/em\u003e\u003ca href=\"#_ENREF_57\" title=\"Kokhlikyan, 2020 #52\"\u003e\u003csup\u003e57\u003c/sup\u003e\u003c/a\u003e and saved to run history log. The run is then repeated until convergence is achieved, or until a pre-determined number of iterations are performed by shuffling training testing and validation data. Results are summarised in a report with exportable tables and graphics. Attribution is applied to true positives and true negatives, and these are translated back to features to be added to reports. Further details can be directly found in the accompanying code\u003ca href=\"#_ENREF_50\" title=\"Ersavas, 2023 #10\"\u003e\u003csup\u003e50\u003c/sup\u003e\u003c/a\u003e.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-3868861/v1/43d1ed913aa2803c3fe43750.png"},{"id":49895706,"identity":"9a1a3bff-f1b6-4b00-81c6-53cfcf43c807","added_by":"auto","created_at":"2024-01-19 21:44:32","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":443257,"visible":true,"origin":"","legend":"\u003cp\u003eA sample from MNIST dataset (left side of each image) and its shuffled counterpart (right side).\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-3868861/v1/5f79ab08e7264c73eb8d16c2.png"},{"id":49895707,"identity":"58e6d8a0-cb44-418f-851d-182bc0991ea6","added_by":"auto","created_at":"2024-01-19 21:44:33","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":238575,"visible":true,"origin":"","legend":"\u003cp\u003eResults of training MNIST dataset (\u003cstrong\u003ea\u003c/strong\u003e) and the shuffled dataset (\u003cstrong\u003eb\u003c/strong\u003e) with PyTorch model \u003cem\u003eResNet18\u003c/em\u003e\u003ca href=\"#_ENREF_50\" title=\"Ersavas, 2023 #10\"\u003e\u003csup\u003e50\u003c/sup\u003e\u003c/a\u003e.\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-3868861/v1/977e36eb046159e47ebf0bc6.png"},{"id":49896227,"identity":"97936ab5-b757-4948-9ed9-60f68d4523fb","added_by":"auto","created_at":"2024-01-19 22:00:33","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":536245,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eDeepMapper \u003c/em\u003ecan find the small variations in very few (in this example two) variables out of very large number of random variables (here 18,225). \u003cstrong\u003ea\u003c/strong\u003e. \u003cem\u003eDeepMapper\u003c/em\u003erepresentations of each record. \u003cstrong\u003eb\u003c/strong\u003e. The result of the test run of the classification with unseen data (3,750 elements). \u003cstrong\u003ec\u003c/strong\u003e. The first and second variables in the graph are measurably higher than the other variables.\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-3868861/v1/8f6206657515e6b899dbaf71.png"},{"id":49896145,"identity":"e50ad8a2-33a9-4836-914d-5fcaf84f24f1","added_by":"auto","created_at":"2024-01-19 21:52:33","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":1098644,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eDeepInsight \u003c/em\u003et-SNE vs\u003cem\u003e DeepMapper \u003c/em\u003eImages. The image on the left was generated by \u003cem\u003eDeepInsight\u003c/em\u003e using its default values and a t-SNE transformer supplied by \u003cem\u003eDeepInsight\u003c/em\u003e. The image on the right was generated by \u003cem\u003eDeepMapper\u003c/em\u003e.\u003c/p\u003e","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-3868861/v1/1717cdc342bf34b715ac1955.png"},{"id":56190199,"identity":"a2bb44dc-2cf4-45d9-9301-77f0adfa0c11","added_by":"auto","created_at":"2024-05-09 16:26:11","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2763011,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3868861/v1/5e3b7ffb-9865-4b1f-8b71-cb60ff85bc0c.pdf"},{"id":49895711,"identity":"1966da79-ab27-4a1e-bd90-b6d1cdf07245","added_by":"auto","created_at":"2024-01-19 21:44:33","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":163846,"visible":true,"origin":"","legend":"","description":"","filename":"Table1and2.docx","url":"https://assets-eu.researchsquare.com/files/rs-3868861/v1/efae238b94353990dd1b711c.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Novel applications of Convolutional Neural Networks in the age of Transformers","fulltext":[{"header":"Introduction","content":"\u003cp\u003eThere are exponential increases in data\u003csup\u003e1\u003c/sup\u003e especially from highly complex systems, whose non-linear interactions and relationships are not well understood, and which can display major or unexpected changes in response to small perturbations, known as the \u0026lsquo;Butterfly effect\u0026rsquo;\u003csup\u003e2\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eIn domains characterised by high-dimensional data, traditional statistical methods and Machine Learning (ML) techniques make heavy use of feature engineering that incorporates extensive filtering, selection of highly variable parameters, and dimension reduction techniques such as Principal Component Analysis (PCA)\u003csup\u003e3\u003c/sup\u003e. Most current tools filter out smaller changes in data, mostly considered artefacts or `noise`, which may contain information that is paramount to understanding the nature and behaviour of such highly complex systems\u003csup\u003e4\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eThe emergence of Deep Learning (DL) offers a paradigm shift. DL algorithms, underpinned by adaptive learning mechanisms, can discern both linear and non-linear data intricacies, and open avenues to analyse data that is not possible or practical by conventional techniques\u003csup\u003e5\u003c/sup\u003e, particularly in complex domains such as image, temporal sequence analysis, molecular biology, and astronomy\u003csup\u003e6\u003c/sup\u003e. DL models, such as \u003cem\u003eConvolutional Neural Networks\u003c/em\u003e (CNNs)\u003csup\u003e7\u003c/sup\u003e, \u003cem\u003eRecurrent Neural Networks\u003c/em\u003e (RNNs)\u003csup\u003e8\u003c/sup\u003e, \u003cem\u003eGenerative\u003c/em\u003e \u003cem\u003eNetwork\u003c/em\u003es\u003csup\u003e9\u003c/sup\u003e and \u003cem\u003eTransformers\u003c/em\u003e\u003csup\u003e10\u003c/sup\u003e, have demonstrated exceptional performance in various domains, such as image and speech recognition, natural language processing, and game playing\u003csup\u003e6\u003c/sup\u003e. CNNs and LSTMs were found to be great tools to predict behaviour of so called `chaotic` systems\u003csup\u003e11\u003c/sup\u003e. Modern DL systems often surpass human-level performance, and challenge humans even in creative endeavours.\u003c/p\u003e\n\u003cp\u003eCNNs utilise a unique architecture that comprises several layers, including convolutional layers, pooling layers, and fully connected layers, to process and transform the input data hierarchically\u003csup\u003e5\u003c/sup\u003e. CNNs have no knowledge of sequence, and therefore are generally not used in analysing time-series or similar data, which is traditionally attempted with Recurrent Neural Networks (RNNs)\u003csup\u003e12\u003c/sup\u003e and Long Short-Term Memory networks (LSTMs)\u003csup\u003e8\u003c/sup\u003e due to their ability to capture temporal patterns. Where CNNs have been employed for sequence or time-series analysis, 1-dimensional (1D) CNNs have been selected because of their vector based 1D input structure\u003csup\u003e13\u003c/sup\u003e. However, attempts to analyse such data in 1D CNNs do not always give superior results\u003csup\u003e14\u003c/sup\u003e. In addition, GPU (Graphical Processing Units) systems are not always optimised for processing 1D CNNs, therefore even though 1D CNNs have fewer parameters than 2-dimensional (2D) CNNs, 2D CNNs can outperform 1D CNNs\u003csup\u003e15\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eTransformers\u003c/em\u003e, introduced by Vaswani and others\u003csup\u003e10\u003c/sup\u003e, have recently come to prominence, particularly for tasks where data is in the form of time series or sequences, in domains ranging from language modelling to stock market prediction\u003csup\u003e16\u003c/sup\u003e. \u003cem\u003eTransformers\u003c/em\u003e leverage self-attention, a key component that allows a model to weigh and focus on various parts of an input sequence when producing an output, enabling the capture of long-range dependencies in data. Unlike CNNs, which use local receptive fields, self-attention weighs the significance of various parts of the input data\u003csup\u003e17\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eFollowing success with sequence-based tasks, \u003cem\u003eTransformers\u003c/em\u003e are being extended to image processing. \u003cem\u003eVision-Transformers\u003c/em\u003e in object detection\u003csup\u003e18\u003c/sup\u003e, \u003cem\u003eDetection Transformers\u003c/em\u003e\u003csup\u003e19\u003c/sup\u003e and lately \u003cem\u003eReal-time Detection Transformers\u003c/em\u003e all claim superiority over CNNs\u003csup\u003e20\u003c/sup\u003e. However, their inference operations demand far more resources than CNNs and trail CNNs in flexibility. They also suffer similar augmentation problems as CNNs. More recently, \u003cem\u003eRetentive-Networks\u003c/em\u003e have been offered as an alternative to \u003cem\u003eTransformers\u003c/em\u003e\u003csup\u003e21\u003c/sup\u003e and may soon challenge the \u003cem\u003eTransformer\u003c/em\u003e architecture.\u003c/p\u003e\n\u003cp\u003eCNNs can recognise dispersed patterns\u003c/p\u003e\n\u003cp\u003eEven though CNNs are widely used, there are some misconceptions, notably that CNNs are largely limited to image data, and require established spatial relationships between pixels in images, both of which are open to challenge. The latter is of particular importance when considering the potential of CNNs to analyse complex non-image datasets, whose data structures are arbitrary.\u003c/p\u003e\n\u003cp\u003eMoreover, while CNNs are universal function approximators\u003csup\u003e22\u003c/sup\u003e, they may not always generalise\u003csup\u003e23\u003c/sup\u003e, especially if they are trained on data that is insufficient to cover the solution space\u003csup\u003e24\u003c/sup\u003e. It is also known that they can spontaneously generalise even when supplied with a small number of samples during training after overfitting, called `grokking`\u003csup\u003e25\u003c/sup\u003e\u003csup\u003e,\u003c/sup\u003e\u003csup\u003e26\u003c/sup\u003e. CNNs can generalise from scattered data if given enough samples, or if they grok, and this can be determined by observing changes to training versus testing accuracy and loss.\u003c/p\u003e\n\u003cp\u003eNon-image processing with CNNs\u003c/p\u003e\n\u003cp\u003eWhile CNNs have achieved remarkable success in computer vision applications, such as image classification and object detection\u003csup\u003e7\u003c/sup\u003e\u003csup\u003e,\u003c/sup\u003e\u003csup\u003e27\u003c/sup\u003e, they have also been employed in other domains to a lesser degree with impressive results, including: (i) natural language processing, text classification, sentiment analysis and named entity recognition, by treating text data as a one-dimensional image with characters represented as pixels\u003csup\u003e16\u003c/sup\u003e\u003csup\u003e,\u003c/sup\u003e\u003csup\u003e28\u003c/sup\u003e; (ii) audio processing, such as speech recognition, speaker identification and audio event detection, by applying convolutions over time frequency representations of audio signals\u003csup\u003e29\u003c/sup\u003e; (iii) time series analysis, such as financial market prediction, human activity recognition and medical signal analysis, using one-dimensional convolutions to capture local temporal patterns and learn features from time series data\u003csup\u003e30\u003c/sup\u003e; and (iv) biopolymer (e.g., DNA) sequencing, using 2D CNNs to accurately classify molecular barcodes in raw signals from Oxford Nanopore sequencers using a transformation to turn a 1D signal into 2D images - improving barcode identification recovery from 38% to over 85%\u003csup\u003e31\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eIndeed, CNNs are not perfect tools for image processing as they do not develop semantic understanding of images even though they can be trained to do semantic segmentation\u003csup\u003e32\u003c/sup\u003e. They cannot easily recognise negative images when trained with positive images\u003csup\u003e33\u003c/sup\u003e. CNNs are also sensitive to the orientation and scale of objects and must rely on augmentation of image datasets, often involving hundreds of variations of the same image\u003csup\u003e34\u003c/sup\u003e. There are no such changes in the perspective and orientation of data converted into flat 2D images.\u003c/p\u003e\n\u003cp\u003eIn the realm of complex domains that generate huge amounts of data, augmentation is usually not required for non-image datasets, as the datasets will be rich enough. Moreover, introducing arbitrary augmentation does not always improve accuracy, indeed, introducing hand-tailored augmentation may hinder analysis\u003csup\u003e35\u003c/sup\u003e. If augmentation is required, it can be introduced in a data-oriented form, but even when using automated augmentation such as \u003cem\u003eAutoAugment\u003csup\u003e35\u003c/sup\u003e\u003c/em\u003e or \u003cem\u003eFasterAutoAugment\u003csup\u003e36\u003c/sup\u003e\u003c/em\u003e, many of the augmentations (such as shearing, translation, rotation, inversion, etc.) should not be used, and the result should be tested carefully, as augmentation may introduce artefacts.\u003c/p\u003e\n\u003cp\u003eA frequent problem with handling non-image datasets with many variables is noise. Many algorithms have been developed for noise elimination, most of which are domain specific. CNNs can be trained to use the whole input space with minimal filtering and no dimension reduction, and can find useful information in what might be ascribed as \u0026lsquo;noise\u0026rsquo;\u003csup\u003e4\u003c/sup\u003e\u003csup\u003e,\u003c/sup\u003e\u003csup\u003e37\u003c/sup\u003e. Indeed, a key reason to retain \u0026lsquo;noise\u0026rsquo; is to allow discovery of small perturbations that cannot be detected by other methods\u003csup\u003e11\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eConversion of non-image data to artificial images for CNN processing\u003c/p\u003e\n\u003cp\u003eTransforming sequence data to images without resorting to dimension reduction or filtering offers a potent toolset for discerning complex patterns in time series and sequence data, which potentiates the two major advantages of CNNs compared to RNNs, LSTMs and \u003cem\u003eTransformers\u003c/em\u003e. First, CNNs do not depend on past data to recognise current patterns, which increases sensitivity to detect patterns that appear in the beginning of time-series or sequence data. Second, 2D CNNs are better optimised for GPUs and highly parallelizable, and are consequently faster than other current architectures, which accelerates training and inference, while reducing resource and energy consumption during in all phases including image transformation, training, and inference significantly.\u003c/p\u003e\n\u003cp\u003eImage data such as MNIST represented in a matrix can be classified by basic deep networks such as \u003cem\u003eMulti-level Perceptrons\u003c/em\u003e (MLP) by turning their matrix representation to vectors (Fig. 1a). Using this approach analysis of images becomes increasingly complex as the image size grows, increasing the input parameters of MLP and the computational cost exponentially. On the other hand, 2D CNNs can handle the original matrix much faster than MLP with equal or better accuracy and scale to much larger images.\u003c/p\u003e\n\u003cp\u003eJust like how a simple neural network analyses a 2D image by turning it into a vector, the reciprocal is also true - data in a vector can be converted to a 2D matrix (Fig. 1b). Vectors converted to such matrices form arbitrary patterns that are incomprehensible to human eye. A similar technique for such mapping has also been proposed by Kovelarchuk et al using another algorithm called CPC-R\u003csup\u003e38\u003c/sup\u003e.\u003c/p\u003e\n\n\u003cp\u003e\u003cem\u003e \u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eAttribution\u003c/p\u003e\n\u003cp\u003eAn important aspect of any analysis is to be able to identify those variables that are most important and the degree to which they contribute to a given classification. Identifying these variables is particularly challenging in CNNs due to their complex hierarchical architecture, and many non-linear transformations\u003csup\u003e39\u003c/sup\u003e. To address this problem many \u0026lsquo;attribution methods\u0026rsquo; have been developed to try to quantify the contribution of each variable (e.g., pixels in images) to the final output for deep neural networks and CNNs\u003csup\u003e40\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eSaliency maps serve as an intuitive attribution and visualisation tool for CNNs, spotlighting regions in input data that significantly influence the model\u0026apos;s predictions\u003csup\u003e27\u003c/sup\u003e. By offering a heatmap representation, these maps illuminate key features that the model deems crucial, thus aiding in demystifying the model\u0026apos;s decision-making process. For instance, when analysing an image of a cat, the saliency map would emphasise the cat\u0026apos;s distinct features over the background. While their simplicity facilitates understanding even for those less acquainted with deep learning, saliency maps do face challenges, particularly their sensitivity to noise and occasional misalignment with human intuition\u003csup\u003e41-43\u003c/sup\u003e. Nonetheless, they remain a pivotal tool in enhancing model transparency and bridging the interpretability gap between ML models and human comprehension.\u003c/p\u003e\n\u003cp\u003eSeveral methods have been proposed for attribution, including \u003cem\u003eGuided Backpropagation\u003c/em\u003e\u003csup\u003e44\u003c/sup\u003e, \u003cem\u003eLayer-wise Relevance Propagation\u003c/em\u003e\u003csup\u003e45\u003c/sup\u003e, \u003cem\u003eGradient-weighted Class Activation Mapping\u003c/em\u003e\u003csup\u003e46\u003c/sup\u003e, \u003cem\u003eIntegrated Gradients\u003c/em\u003e\u003csup\u003e47\u003c/sup\u003e, \u003cem\u003eDeepLIFT\u003c/em\u003e\u003csup\u003e48\u003c/sup\u003e, and \u003cem\u003eSHAP\u003c/em\u003e (SHapley Additive exPlanations)\u003csup\u003e49\u003c/sup\u003e. Many of these methods were developed because it is challenging to identify important input features when there are different images with the same label (e.g., \u0026lsquo;bird\u0026rsquo; with many species) presented at different scales, colours, and perspectives. In contrast, most non-image data does not have such variations, as each pixel corresponds to the same feature. For this reason, choosing attributions with minimal processing is sufficient to identify the salient input variables that have the maximal impact on classification.\u003cem\u003e\u003c/em\u003e\u003cem\u003e \u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eDeepMapper\u003c/p\u003e\n\u003cp\u003eHere we introduce a new analytical pipeline, \u003cem\u003eDeepMapper\u003c/em\u003e, which applies a non-indexed or indexed mapping to the data representing each data point with one pixel, enabling the classification or clustering of data using 2D CNNs. This simple direct mapping has been tried by others but has not been tested with datasets with sufficiently large amounts of data in various conditions.\u003c/p\u003e\n\u003cp\u003eThe pipeline includes conversion of data, separation to training and validation, assessment of training quality, attribution, and accumulation of results in a pipeline. The pipeline is run multiple times until a consensus is reached. The significant variables can then be identified using attribution and exported appropriately.\u003c/p\u003e\n\u003cp\u003eThe \u003cem\u003eDeepMapper \u003c/em\u003earchitecture is shown in Fig. 2. The complete algorithm of \u003cem\u003eDeepMapper \u003c/em\u003eis detailed in the methods section and the Python source code is supplied at GitHub\u003csup\u003e50\u003c/sup\u003e.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eWe present a simple example to demonstrate that CNNs can readily interpret data with a well dispersed pattern of pixels, using the MNIST dataset, which is widely used for hand-written image recognition and which humans as well as CNNs can easily recognise and classify based on the obvious spatial relationships between pixels (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). This dataset is a more complicated problem than datasets such as the \u003cem\u003eGisette\u003c/em\u003e dataset\u003csup\u003e\u003cspan citationid=\"CR51\" class=\"CitationRef\"\u003e51\u003c/span\u003e\u003c/sup\u003e that was developed to distinguish between 4 and 9. It includes all digits and uses a full randomisation of pixels, and can be regenerated with the script supplied\u003csup\u003e\u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e\u003c/sup\u003e and changing the seed will generate different patterns.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eWe randomly shuffled the data in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e using the same seed\u003csup\u003e\u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e\u003c/sup\u003e to obtain 60,000 training images such as those shown on the right side of each digit, and validated the results with a separate batch of 20,000 images (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). Although the resulting images are no longer recognizable by eye, a CNN has no difficulty distinguishing and classifying each pattern with ~\u0026thinsp;2% testing error compared to the reference data (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e). This result demonstrates that CNNs can accurately recognise global patterns in images without reliance on local relationships between neighbouring pixels. It also confirms the finding that shuffling images only marginally increases training loss\u003csup\u003e\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e and extends it to testing loss (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eTesting DeepMapper\u003c/h2\u003e \u003cp\u003eFinding slight changes in very few variables in otherwise seemingly random datasets with large numbers of variables is like finding a needle in a haystack. Such differences in data are almost impossible to detect using traditional analysis tools because small variations are usually filtered out before analysis.\u003c/p\u003e \u003cp\u003eWe devised a simple test case to determine if \u003cem\u003eDeepMapper\u003c/em\u003e can detect one or more variables with small but distinct variations in otherwise randomly generated data. We generated two sets of 5,000 data items with 18,225 numeric variables. Using PyTorch\u0026rsquo;s uniform random algorithms\u003csup\u003e\u003cspan citationid=\"CR52\" class=\"CitationRef\"\u003e52\u003c/span\u003e\u003c/sup\u003e that set 17,998 of these to random numbers in the range of 0\u0026ndash;1, and two of the variables into two distinct groups as seen in Table\u0026nbsp;1.\u003c/p\u003e \u003cp\u003eWe call this type of dataset \u0026lsquo;Needle in a haystack\u0026rsquo; (NIHS) dataset, where very small amounts of data with small variance is hidden among a set of random variables that is order(s) of magnitude greater than the meaningful components. We provide a script that can generate this and similar datasets among the source supplied\u003csup\u003e\u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003e \u003cem\u003eDeepMapper\u003c/em\u003e was able to accurately classify the two datasets (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e). Furthermore, using attribution \u003cem\u003eDeepMapper\u003c/em\u003e was also able to determine the two datapoints that have small but different variances in the two classes. Note that \u003cem\u003eDeepMapper\u003c/em\u003e may not always find all the changes as neural network initialisation of weights is a stochastic process. However, \u003cem\u003eDeepMapper o\u003c/em\u003evercomes this matter via multiple iterations to establish acceptable training and testing accuracies as described in the Methods.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003eComparison of DeepMapper with DeepInsight\u003c/h2\u003e \u003cp\u003e \u003cem\u003eDeepInsight\u003c/em\u003e is the most general approach published to date for converting non-image data into image-like structures, with the claim that these processed structures allow CNNs to capture complex patterns and features in the data\u003csup\u003e\u003cspan citationid=\"CR53\" class=\"CitationRef\"\u003e53\u003c/span\u003e\u003c/sup\u003e. \u003cem\u003eDeepInsight\u003c/em\u003e offers an algorithm to create images that have similar features collated into a \u0026ldquo;well organised image form\u0026rdquo;, or by applying one of several dimensionality reduction algorithms (e.g., t-SNE, PCA or KPCA)\u003csup\u003e\u003cspan citationid=\"CR53\" class=\"CitationRef\"\u003e53\u003c/span\u003e\u003c/sup\u003e. However, these algorithms add computational complexity, potentially eliminate valuable information, limit the abilities of CNNs to find small perturbations, and make it more difficult to use attribution to determine most notable features impacting analysis as multiple features may overlap in the transformed image. In contrast \u003cem\u003eDeepMapper\u003c/em\u003e uses a direct mapping mechanism where each feature corresponds to one pixel.\u003c/p\u003e \u003cp\u003eTo identify important input variables, \u003cem\u003eDeepInsight\u003c/em\u003e incorporates \u003cem\u003eDeepFeature\u003c/em\u003e\u003csup\u003e\u003cspan citationid=\"CR54\" class=\"CitationRef\"\u003e54\u003c/span\u003e\u003c/sup\u003e using an elaborate mechanism to associate image areas identified by attribution methods to the input variables. \u003cem\u003eDeepMapper\u003c/em\u003e uses a simpler approach as each pixel corresponds to only one variable and can use any of the attribution methods to link results to its input space. While both \u003cem\u003eDeepMapper\u003c/em\u003e and \u003cem\u003eDeepInsight\u003c/em\u003e follow the general idea that non-image data can be processed with 2D CNNs, \u003cem\u003eDeepMapper\u003c/em\u003e uses a much simpler and faster algorithm, while \u003cem\u003eDeepInsight\u003c/em\u003e chooses a sophisticated set of algorithms to convert non-image data to images, dramatically increasing computational cost. The \u003cem\u003eDeepInsight\u003c/em\u003e conversion process is not designed to utilise GPUs so cannot be accelerated by better hardware, and the obtained images may be larger than the number of data points, also impacting performance.\u003c/p\u003e \u003cp\u003eThe \u003cem\u003eDeepInsight\u003c/em\u003e manuscript offers various examples of data to demonstrate its abilities. However, many of the examples use low dimensions (20 to ~\u0026thinsp;4,000 features) while today\u0026rsquo;s complex datasets may regularly require tens of thousands to millions of features such as in genome analysis in biology and radio-telescope analysis in astronomy. As such, several examples provided by \u003cem\u003eDeepInsight\u003c/em\u003e have insufficient dimensions for a sophisticated mechanism such as \u003cem\u003eDeepMapper\u003c/em\u003e, which should ideally have 10,000 or more dimensions as required by modern complex datasets. \u003cem\u003eDeepInsight\u003c/em\u003e examples include a speech dataset from the TIMIT corpus with 39 dimensions, \u003cem\u003eRelathe\u003c/em\u003e (text) dataset, which is derived from newsgroup documents and partitioned evenly across different newsgroups. It contains 1,427 samples and 4,322 dimensions. The \u003cem\u003eringnorm-DELVE\u003c/em\u003e, which is an implementation of Leo Breiman\u0026rsquo;s ringnorm example, is a 20 dimensional, 2 class classification with 7,400 samples\u003csup\u003e\u003cspan citationid=\"CR53\" class=\"CitationRef\"\u003e53\u003c/span\u003e\u003c/sup\u003e. Another example, \u003cem\u003eMadelon\u003c/em\u003e, introduced an artificially generated dataset 2,600 samples and 500 dimensions, where only 5 principal and 20 derived variables containing information. Instead, we used a much more complicated example than \u003cem\u003eMadelon\u003c/em\u003e, an NIHS dataset\u003csup\u003e\u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e\u003c/sup\u003e that we used to test \u003cem\u003eDeepMapper\u003c/em\u003e in the first place. We attempted to run \u003cem\u003eDeepInsight\u003c/em\u003e with NIHS data, but we could not get it to train properly and for this reason we cannot supply a comparison.\u003c/p\u003e \u003cp\u003eThe most complex problem analysed by \u003cem\u003eDeepInsight\u003c/em\u003e was a public RNA sequencing gene expression dataset from TCGA (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://cancergenome.nih.gov/\u003c/span\u003e\u003cspan address=\"https://cancergenome.nih.gov/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e) containing 6,216 samples of 60,483 genes or dimensions. \u003cem\u003eDeepMapper\u003c/em\u003e exhibited much more improved processing speed with near identical accuracy (Table\u0026nbsp;2, Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eCNNs are fundamentally sophisticated pattern matchers that can establish intricate mappings between input features and output representations\u003csup\u003e6\u003c/sup\u003e. They excel at transforming various inputs into outputs, including identifying classes or bounding boxes, through a series of operations involving convolution, pooling, and activation functions\u003csup\u003e7\u003c/sup\u003e\u003csup\u003e,\u003c/sup\u003e\u003csup\u003e55\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eEven though CNNs are in the centre of many of today\u0026rsquo;s revolutionary AI systems from self-driving cars to generative AI systems such as \u003cem\u003eDall-E-2\u003c/em\u003e, \u003cem\u003eMidJourney\u003c/em\u003e and \u003cem\u003eStable Diffusion\u003c/em\u003e, they are still not well understood nor efficiently utilised, and their usage beyond image analysis has been limited.\u003c/p\u003e\n\u003cp\u003eWhile CNNs used in image analysis are constrained historically and practically to a 224 x 224 matrix or a similar fixed size input, this limitation arises for pre-trained models. When CNNs have not been pre-trained, one can select a much wider variety of sizes as input shape depending on the CNN architecture. Some CNNs are more flexible in their input size that implemented with adaptive pooling layers such as ResNet18 using adaptive pooling\u003csup\u003e56\u003c/sup\u003e. This provides flexibility to choose optimal sizes for the task in hand for non-image applications, as most non-image applications will not use pre-trained CNNs.\u003c/p\u003e\n\u003cp\u003eHere we have demonstrated uses of CNNs that are outside the norm. There is a need for analysis of complex data with many thousands of features that are not primarily images. There is also a lack of tools that offer minimal conversion of non-image data to image-like formats that then can easily be processed with CNNs in classification and clustering tasks. As a lot of this data is coming from complex systems that have a lot of features, \u003cem\u003eDeepMapper\u003c/em\u003e offers a way of investigating such data in ways that may not be possible with traditional approaches. \u003c/p\u003e\n\u003cp\u003eEven though \u003cem\u003eDeepMapper\u003c/em\u003e currently uses CNN as its AI component, alternative analytic strategies can easily be substituted in lieu of CNN with minimal changes, such as \u003cem\u003eVision Transformers\u003c/em\u003e\u003csup\u003e18\u003c/sup\u003e or \u003cem\u003eRetNets\u003c/em\u003e\u003csup\u003e21\u003c/sup\u003e, which have great potential for this application. While \u003cem\u003eTransformers\u003c/em\u003e and \u003cem\u003eRetNets\u003c/em\u003e have input size limitations for inference in terms of number of tokens. \u003cem\u003eVision Transformers\u003c/em\u003e can handle much larger inputs by dividing images to segments that incorporate multiple pixels\u003csup\u003e18\u003c/sup\u003e. This type of approach would be applicable to both \u003cem\u003eTransformers\u003c/em\u003e and \u003cem\u003eRetNets\u003c/em\u003e, and future architectures. \u003cem\u003eDeepMapping\u003c/em\u003e can leverage these newer architectures, and others, in the future\u003csup\u003e56\u003c/sup\u003e.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003e\u003cem\u003eDeepMapper\u003c/em\u003e is developed to implement an approach to process high-dimensional data without resorting to excessive filtering and dimension reduction techniques that eliminate smaller perturbations in data to be able to identify those differences that would otherwise be filtered out. The following algorithm is used to achieve this result:\u003c/p\u003e\n\u003col\u003e\n \u003cli\u003eRead and setup the running parameters.\u003c/li\u003e\n \u003cli\u003eRead the data into a tabulated form in the form of observations, features, and outcome (in the form of labels, or if self-supervised, the input itself).\u003c/li\u003e\n \u003cli\u003eIdentify features and labels.\u003c/li\u003e\n \u003cli\u003eDo only basic filtering that eliminates observations or features if all of them are 0 or empty.\u003c/li\u003e\n \u003cli\u003eNormalise features.\u003c/li\u003e\n \u003cli\u003eTransform tabulated data to tensors of rank 2.\u003c/li\u003e\n \u003cli\u003eIf the analysis is supervised, then transform labels to output tensors.\u003c/li\u003e\n \u003cli\u003eBegin iteration:\u003col style=\"list-style-type: lower-alpha;\"\u003e\n \u003cli\u003eSeparate the data into training and testing, and validation groups.\u003c/li\u003e\n \u003cli\u003eTrain on the dataset for required number of epochs, until reaching satisfactory testing accuracy and loss, or maximum a pre-determined number of iterations.\u003c/li\u003e\n \u003cli\u003eIf satisfactory testing results are obtained, then:\u003col style=\"list-style-type: lower-roman;\"\u003e\n \u003cli\u003ePerform attributions.\u003c/li\u003e\n \u003cli\u003eAccumulate attribution results.\u003c/li\u003e\n \u003c/ol\u003e\n \u003c/li\u003e\n \u003cli\u003eIf training is satisfactory:\u003col style=\"list-style-type: lower-roman;\"\u003e\n \u003cli\u003eTabulate attribution results.\u003c/li\u003e\n \u003cli\u003eSave the model.\u003c/li\u003e\n \u003c/ol\u003e\n \u003c/li\u003e\n \u003c/ol\u003e\n \u003c/li\u003e\n \u003cli\u003eReport results.\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThe results of \u003cem\u003eDeepMapper\u003c/em\u003e analysis can be used in 2 ways:\u003c/p\u003e\n\u003col\u003e\n \u003cli\u003eSupervised: \u003cem\u003eDeepMapper\u003c/em\u003e produces a list of features that played a prominent role in the differentiation of classes\u003c/li\u003e\n \u003cli\u003eSelf-supervised: Highlight features that are the most important features in differentiating observations from each other in a non-linear fashion. The output can be used as an alternative feature selection tool for dimension reduction\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eIn both modes, any hidden layer can be examined as latent space. A special bottleneck layer can be introduced to reduce dimensions for clustering purposes.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAcknowledgements\u003c/h2\u003e\n\u003cp\u003eWe thank\u0026nbsp;Murat Karaorman,\u0026nbsp;Mitchell Cummins, and Fatemeh Vafaee for helpful advice and comments on the manuscript. This research is supported by an Australian Government Research Training Program Scholarships RSAI8000 and RSAP1000 to T.E., a\u0026nbsp;Fonds de Recherche du Quebec Sant\u0026eacute; Junior 1 Award 284217 to M.A.S., and UNSW SHARP grant RG193211 to J.S.M.\u003c/p\u003e\n\u003ch2\u003eAuthor contributions\u003c/h2\u003e\n\u003cp\u003eT.E. developed the methods, implemented \u003cem\u003eDeepMapper\u0026nbsp;\u003c/em\u003eand produced the first draft of the paper. J.S.M. provided advice, structured the paper, and edited it for improved readability and clarity. M.A.S. provided advice and edited the paper.\u003c/p\u003e\n\u003ch2\u003eCompeting interests\u003c/h2\u003e\n\u003cp\u003eAuthors claim no competing interests.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eTaylor, P. Volume of data/information created, captured, copied, and consumed worldwide from 2010 to 2020, with forecasts from 2021 to 2025. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.statista.com/statistics/871513/worldwide-data-created/\u003c/span\u003e\u003cspan address=\"https://www.statista.com/statistics/871513/worldwide-data-created/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGhys, \u0026Eacute;. The butterfly effect. in \u003cem\u003eThe Proceedings of the 12th International Congress on Mathematical Education: Intellectual and attitudinal challenges\u003c/em\u003e, pp. 19\u0026ndash;39 (Springer). https://doi.org/20.500.12657/28000/1/1001997 (2015).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJolliffe, I. T. Mathematical and statistical properties of sample principal components. \u003cem\u003ePrincipal Component Analysis\u003c/em\u003e, pp. 29\u0026ndash;61 (Springer). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/0-387-22440-8_3\u003c/span\u003e\u003cspan address=\"10.1007/0-387-22440-8_3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2002).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLandauer, R. The noise is the signal. \u003cem\u003eNature\u003c/em\u003e 392, 658\u0026ndash;659. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/33551\u003c/span\u003e\u003cspan address=\"10.1038/33551\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (1998).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGoodfellow, I., Bengio, Y. \u0026amp; Courville, A. \u003cem\u003eDeep Learning\u003c/em\u003e (MIT Press). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.deeplearningbook.org\u003c/span\u003e\u003cspan address=\"http://www.deeplearningbook.org\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLeCun, Y., Bengio, Y. \u0026amp; Hinton, G. Deep learning. \u003cem\u003eNature\u003c/em\u003e 521, 436\u0026ndash;444. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/nature14539\u003c/span\u003e\u003cspan address=\"10.1038/nature14539\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2015).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKrizhevsky, A., Sutskever, I. \u0026amp; Hinton, G. E. ImageNet classification with deep convolutional neural networks. \u003cem\u003eCommunications of the ACM\u003c/em\u003e 60, 84\u0026ndash;90. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://dl.acm.org/doi/10.1145/3065386\u003c/span\u003e\u003cspan address=\"https://dl.acm.doi/10.1145/3065386\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHochreiter, S. \u0026amp; Schmidhuber, J. Long short-term memory. \u003cem\u003eNeural Computation\u003c/em\u003e 9, 1735\u0026ndash;1780. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1162/neco.1997.9.8.1735\u003c/span\u003e\u003cspan address=\"10.1162/neco.1997.9.8.1735\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (1997).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGoodfellow, I. \u003cem\u003eet al.\u003c/em\u003e Generative adversarial nets. \u003cem\u003eCommunications of the ACM\u003c/em\u003e 63, 139\u0026ndash;144. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://dl.acm.org/doi/10.1145/3422622\u003c/span\u003e\u003cspan address=\"https://dl.acm.doi/10.1145/3422622\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVaswani, A. \u003cem\u003eet al.\u003c/em\u003e Attention is all you need. \u003cem\u003eNIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems\u003c/em\u003e, pp. 6000\u0026ndash;6010. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://dl.acm.org/doi/10.5555/3295222.3295349\u003c/span\u003e\u003cspan address=\"https://dl.acm.doi/10.5555/3295222.3295349\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBarrio, R. \u003cem\u003eet al.\u003c/em\u003e Deep learning for chaos detection. \u003cem\u003eChaos\u003c/em\u003e 33, article 073146. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1063/5.0143876\u003c/span\u003e\u003cspan address=\"10.1063/5.0143876\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLevin, E. A recurrent neural network: limitations and training. \u003cem\u003eNeural Networks\u003c/em\u003e 3, 641\u0026ndash;650. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/0893-6080(90)90054-O\u003c/span\u003e\u003cspan address=\"10.1016/0893-6080(90)90054-O\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (1990).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLeCun, Y. \u0026amp; Bengio, Y. Convolutional networks for images, speech, and time series. in \u003cem\u003eThe handbook of brain theory and neural networks\u003c/em\u003e, pp. 255\u0026ndash;258. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://dl.acm.org/doi/10.5555/303568.303704\u003c/span\u003e\u003cspan address=\"https://dl.acm.doi/10.5555/303568.303704\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (MIT Press) 1998).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWu, Y., Yang, F., Liu, Y., Zha, X. \u0026amp; Yuan, S. A comparison of 1-D and 2-D deep convolutional neural networks in ECG classification. \u003cem\u003earXiv preprint arXiv:1810.07088\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.1810.07088\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.1810.07088\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHu, J. \u003cem\u003eet al.\u003c/em\u003e A multichannel 2D convolutional neural network model for task-evoked fMRI data classification. \u003cem\u003eComputational Intelligence and Neuroscience\u003c/em\u003e 2019, article 5065214. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1155/2019/5065214\u003c/span\u003e\u003cspan address=\"10.1155/2019/5065214\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang, S. \u003cem\u003eet al.\u003c/em\u003e A deep learning framework for modeling structural features of RNA-binding protein targets. \u003cem\u003eNucleic Acids Research\u003c/em\u003e 44, e32. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/nar/gkv1025\u003c/span\u003e\u003cspan address=\"10.1093/nar/gkv1025\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMaur\u0026iacute;cio, J., Domingues, I. \u0026amp; Bernardino, J. Comparing vision transformers and convolutional neural networks for image classification: A literature review. \u003cem\u003eApplied Sciences\u003c/em\u003e 13, article 5521. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/app13095521\u003c/span\u003e\u003cspan address=\"10.3390/app13095521\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDosovitskiy, A. \u003cem\u003eet al.\u003c/em\u003e An image is worth 16x16 words: Transformers for image recognition at scale. \u003cem\u003earXiv preprint arXiv\u003c/em\u003e:2010.\u003cem\u003e11929\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2010.11929\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2010.11929\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCarion, N. \u003cem\u003eet al.\u003c/em\u003e End-to-end object detection with transformers. \u003cem\u003eComputer Vision-ECCV 2020\u003c/em\u003e (Springer), pp. 213\u0026ndash;229. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://link.springer.com/chapter/10.1007/\u003c/span\u003e\u003cspan address=\"https://link.springer.com/chapter/10.1007/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e978-3-030-58452-8_13 (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLv, W. \u003cem\u003eet al.\u003c/em\u003e DETRs beat YOLOs on real-time object detection. \u003cem\u003earXiv preprint arXiv:2304.08069.\u003c/em\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2304.08069\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2304.08069\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSun, Y. \u003cem\u003eet al.\u003c/em\u003e Retentive network: A successor to transformer for large language models. \u003cem\u003earXiv preprint arXiv:2307.08621\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2307.08621\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2307.08621\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhou, D.-X. Universality of deep convolutional neural networks. \u003cem\u003eApplied and Computational Harmonic Analysis\u003c/em\u003e 48, 787\u0026ndash;794. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.acha.2019.06.004\u003c/span\u003e\u003cspan address=\"10.1016/j.acha.2019.06.004\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChiyuan, Z., Bengio, S., Hardt, M., Recht, B. \u0026amp; Vinyals, O. Understanding deep learning (still) requires rethinking generalization. \u003cem\u003eCommunications of the ACM\u003c/em\u003e 64, 107\u0026ndash;115. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3446776\u003c/span\u003e\u003cspan address=\"10.1145/3446776\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMa, W., Papadakis, M., Tsakmalis, A., Cordy, M. \u0026amp; Traon, Y. L. Test selection for deep learning systems. \u003cem\u003eACM Transactions on Software Engineering and Methodology\u003c/em\u003e 30, article 13. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3417330\u003c/span\u003e\u003cspan address=\"10.1145/3417330\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu, Z., Michaud, E. J. \u0026amp; Tegmark, M. Omnigrok: grokking beyond algorithmic data. \u003cem\u003earXiv preprint arXiv:2210.01117.\u003c/em\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2210.01117\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2210.01117\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePower, A., Burda, Y., Edwards, H., Babuschkin, I. \u0026amp; Misra, V. Grokking: generalization beyond overfitting on small algorithmic datasets. \u003cem\u003earXiv preprint arXiv:2201.02177.\u003c/em\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2201.02177\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2201.02177\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSimonyan, K., Vedaldi, A. \u0026amp; Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. \u003cem\u003earXiv preprint arXiv:1312.6034\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.1312.6034\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.1312.6034\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2013).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKim, Y. Convolutional neural networks for sentence classification. \u003cem\u003earXiv preprint arXiv:1408.5882\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.1408.5882\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.1408.5882\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2014).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbdel-Hamid, O. \u003cem\u003eet al.\u003c/em\u003e Convolutional neural networks for speech recognition. \u003cem\u003eIEEE/ACM Transactions on Audio, Speech, and Language Processing\u003c/em\u003e 22, 1533\u0026ndash;1545. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/TASLP.2014.2339736\u003c/span\u003e\u003cspan address=\"10.1109/TASLP.2014.2339736\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2014).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHatami, N., Gavet, Y. \u0026amp; Debayle, J. Classification of time-series images using deep convolutional neural networks. \u003cem\u003eProceedings Tenth International Conference on Machine Vision (ICMV\u003c/em\u003e 2017) 10696, 106960Y. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1117/12.2309486\u003c/span\u003e\u003cspan address=\"10.1117/12.2309486\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSmith, M. A. \u003cem\u003eet al.\u003c/em\u003e Molecular barcoding of native RNAs using nanopore sequencing and deep learning. \u003cem\u003eGenome Research\u003c/em\u003e 30, 1345\u0026ndash;1353. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1101/gr.260836.120\u003c/span\u003e\u003cspan address=\"10.1101/gr.260836.120\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEmek Soylu, B. \u003cem\u003eet al.\u003c/em\u003e Deep-Learning-based approaches for semantic segmentation of natural scene images: a review. \u003cem\u003eElectronics\u003c/em\u003e 12, article 2730. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/electronics12122730\u003c/span\u003e\u003cspan address=\"10.3390/electronics12122730\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHosseini, H., Xiao, B., Jaiswal, M. \u0026amp; Poovendran, R. On the limitation of Convolutional Neural Networks in recognizing negative images. \u003cem\u003e16th IEEE International Conference on Machine Learning and Applications\u003c/em\u003e, pp. 352\u0026ndash;358. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ieeexplore.ieee.org/document/8260656\u003c/span\u003e\u003cspan address=\"https://ieeexplore.ieee.org/document/8260656\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMontserrat, D. M., Lin, Q., Allebach, J. \u0026amp; Delp, E. J. Training object detection and recognition CNN models using data augmentation. \u003cem\u003eElectronic Imaging\u003c/em\u003e 2017, 27\u0026ndash;36. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2352/ISSN.2470-1173.2017.10.IMAWM-163\u003c/span\u003e\u003cspan address=\"10.2352/ISSN.2470-1173.2017.10.IMAWM-163\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCubuk, E. D., Zoph, B., Mane, D., Vasudevan, V. \u0026amp; Le, Q. V. Autoaugment: learning augmentation policies from data. \u003cem\u003earXiv preprint arXiv:1805.09501\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.1805.09501\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.1805.09501\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHataya, R., Zdenek, J., Yoshizoe, K. \u0026amp; Nakayama, H. Faster AutoAugment: Learning augmentation strategies using backpropagation, in \u003cem\u003eComputer Vision\u0026ndash;ECCV 2020: 16th European Conference, Proceedings, Part XXV\u003c/em\u003e, pp. 1\u0026ndash;16 (Springer). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://link.springer.com/chapter/\u003c/span\u003e\u003cspan address=\"https://link.springer.com/chapter/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/978-3-030-58595-2_1\u003c/span\u003e\u003cspan address=\"10.1007/978-3-030-58595-2_1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eXiao, K., Engstrom, L., Ilyas, A. \u0026amp; Madry, A. Noise or signal: the role of image backgrounds in object recognition. \u003cem\u003earXiv preprint arXiv\u003c/em\u003e:2006.\u003cem\u003e09994\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2006.09994\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2006.09994\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKovalerchuk, B., Kalla, D. C. \u0026amp; Agarwal, B., Deep learning image recognition for non-images, in \u003cem\u003eIntegrating artificial intelligence and visualization for visual knowledge discovery\u003c/em\u003e (eds. Kovalerchuk, B., \u003cem\u003eet al.\u003c/em\u003e) pp. 63\u0026ndash;100 (Springer). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/978-3-030-93119-3_3\u003c/span\u003e\u003cspan address=\"10.1007/978-3-030-93119-3_3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSamek, W., Binder, A., Montavon, G., Lapuschkin, S. \u0026amp; Muller, K. R. Evaluating the visualization of what a deep neural network has learned. \u003cem\u003eIEEE Transactions in Neural Networks and Learning Systems\u003c/em\u003e 28, 2660\u0026ndash;2673. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/tnnls.2016.2599820\u003c/span\u003e\u003cspan address=\"10.1109/tnnls.2016.2599820\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMontavon, G., Samek, W. \u0026amp; M\u0026uuml;ller, K.-R. Methods for interpreting and understanding deep neural networks. \u003cem\u003eDigital Signal Processing\u003c/em\u003e 73, 1\u0026ndash;15. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.dsp.2017.10.011\u003c/span\u003e\u003cspan address=\"10.1016/j.dsp.2017.10.011\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDe Cesarei, A., Cavicchi, S., Cristadoro, G. \u0026amp; Lippi, M. Do humans and deep convolutional neural networks use visual information similarly for the categorization of natural scenes? \u003cem\u003eCognitive Science\u003c/em\u003e 45, e13009. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/cogs.13009\u003c/span\u003e\u003cspan address=\"10.1111/cogs.13009\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKindermans, P.-J. \u003cem\u003eet al.\u003c/em\u003e The (un) reliability of saliency methods, in \u003cem\u003eExplainable AI: Interpreting, explaining and visualizing deep learning\u003c/em\u003e. \u003cem\u003eLecture Notes in Computer Science\u003c/em\u003e \u0026lt;background-color:#FFCC66;ivertical-align:baseline;\u0026gt;11700\u0026lt;/background-color:#FFCC66;ivertical-align:baseline;\u0026gt;, pp. 267\u0026ndash;280 (Springer). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/978-3-030-28954-6_14\u003c/span\u003e\u003cspan address=\"10.1007/978-3-030-28954-6_14\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZeiler, M. D. \u0026amp; Fergus, R. Visualizing and understanding convolutional networks. \u003cem\u003eComputer Vision\u003c/em\u003e -- \u003cem\u003eECCV 2014\u003c/em\u003e, pp. 818\u0026ndash;833 (Fleet, D., Pajdla T., Schiele, B., \u0026amp; Tuytelaars, T., eds) (Springer). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/978-3-319-10590-1_53\u003c/span\u003e\u003cspan address=\"10.1007/978-3-319-10590-1_53\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2014).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSpringenberg, J. T., Dosovitskiy, A., Brox, T. \u0026amp; Riedmiller, M. Striving for simplicity: The all convolutional net. \u003cem\u003earXiv preprint arXiv:1412.6806\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.1412.6806\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.1412.6806\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2014).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBinder, A., Montavon, G., Lapuschkin, S., M\u0026uuml;ller, K.-R. \u0026amp; Samek, W. Layer-wise relevance propagation for neural networks with local renormalization layers, in \u003cem\u003eArtificial Neural Networks and Machine Learning\u0026ndash;ICANN 2016: Proceedings 25th International Conference on Artificial Neural Networks\u003c/em\u003e, pp. 63\u0026ndash;71 (Springer). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/978-3-319-44781-0_8\u003c/span\u003e\u003cspan address=\"10.1007/978-3-319-44781-0_8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSelvaraju, R. R. \u003cem\u003eet al.\u003c/em\u003e Grad-cam: visual explanations from deep networks via gradient-based localization. \u003cem\u003eProceedings of the\u003c/em\u003e 2017 \u003cem\u003eIEEE international conference on computer vision\u003c/em\u003e, pp. 618\u0026ndash;626. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ieeexplore.ieee.org/document/8237336\u003c/span\u003e\u003cspan address=\"https://ieeexplore.ieee.org/document/8237336\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSundararajan, M., Taly, A. \u0026amp; Yan, Q. Axiomatic attribution for deep networks. \u003cem\u003eProceedings of the 34th International Conference on Machine Learning\u003c/em\u003e 70, 3319\u0026ndash;3328. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://dl.acm.org/doi/10.5555/3305890.3306024\u003c/span\u003e\u003cspan address=\"https://dl.acm.doi/10.5555/3305890.3306024\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShrikumar, A., Greenside, P. \u0026amp; Kundaje, A. Learning important features through propagating activation differences. \u003cem\u003eProceedings of the 34th International Conference on Machine Learning\u003c/em\u003e 70, 3145\u0026ndash;3153. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://dl.acm.org/doi/10.5555/3305890.3306006\u003c/span\u003e\u003cspan address=\"https://dl.acm.doi/10.5555/3305890.3306006\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLundberg, S. M. \u0026amp; Lee, S.-I. A unified approach to interpreting model predictions. \u003cem\u003eProceedings of the 31st International Conference on Machine Learning\u003c/em\u003e, pp. 4768\u0026ndash;4777. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://dl.acm.org/doi/10.5555/3295222.3295230\u003c/span\u003e\u003cspan address=\"https://dl.acm.doi/10.5555/3295222.3295230\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eErsavas, T. Deepmapper. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/tansel/deepmapper\u003c/span\u003e\u003cspan address=\"https://github.com/tansel/deepmapper\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGuyon, I. G. S. B.-H. A. \u0026amp; Dror, G. Gisette. \u003cem\u003eUCI Machine Learning Repository\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://archive.ics.uci.edu/dataset/170/gisette\u003c/span\u003e\u003cspan address=\"https://archive.ics.uci.edu/dataset/170/gisette\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2008).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePyTorch, torch.rand. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://pytorch.org/docs/stable/generated/torch.rand.html\u003c/span\u003e\u003cspan address=\"https://pytorch.org/docs/stable/generated/torch.rand.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSharma, A., Vans, E., Shigemizu, D., Boroevich, K. A. \u0026amp; Tsunoda, T. DeepInsight: A methodology to transform a non-image data to an image for convolution neural network architecture. \u003cem\u003eScientific Reports\u003c/em\u003e 9, article 11399. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41598-019-47765-6\u003c/span\u003e\u003cspan address=\"10.1038/s41598-019-47765-6\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSharma, A., Lysenko, A., Boroevich, K. A., Vans, E. \u0026amp; Tsunoda, T. DeepFeature: feature selection in nonimage data using convolutional neural network. \u003cem\u003eBriefings in Bioinformatics\u003c/em\u003e 22, bbab297. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/bib/bbab297\u003c/span\u003e\u003cspan address=\"10.1093/bib/bbab297\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSimonyan, K. \u0026amp; Zisserman, A. Very deep convolutional networks for large-scale image recognition. \u003cem\u003earXiv preprint arXiv:1409.1556\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.1409.1556\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.1409.1556\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2014).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePytorch2, AdaptiveAvgPool2d. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html\u003c/span\u003e\u003cspan address=\"https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKokhlikyan, N. \u003cem\u003eet al.\u003c/em\u003e Captum: a unified and generic model interpretability library for PyTorch. \u003cem\u003earXiv preprint arXiv\u003c/em\u003e:2009.\u003cem\u003e07896\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2009.07896\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2009.07896\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2020).\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003eTable 1 and 2 are available in the Supplementary Files section.\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-3868861/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3868861/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eConvolutional Neural Networks (CNNs) have been central to the Deep Learning revolution and played a key role in initiating the new age of Artificial Intelligence. However, in recent years newer architectures such as \u003cem\u003eTransformers\u003c/em\u003e have dominated both research and practical applications. While CNNs still play critical roles in many of the newer developments such as Generative AI, they are far from being thoroughly understood and utilised to their full potential. Here we show that CNNs can recognise patterns in images with scattered pixels and can be used to analyse complex datasets by transforming them into pseudo images in a standardised way for any high dimensional dataset, representing a major advance in the application of CNNs to datasets such as in molecular biology, text, and speech. We introduce a simple approach called \u003cem\u003eDeepMapping\u003c/em\u003e, which allows analysis of very high dimensional datasets without intermediate filtering and dimension reduction, thus preserving the full texture of the data, enabling the ability to detect small perturbations. We also demonstrate that \u003cem\u003eDeepMapper\u003c/em\u003e is superior in speed and on par in accuracy to prior work in processing large datasets with large numbers of features.\u003c/p\u003e","manuscriptTitle":"Novel applications of Convolutional Neural Networks in the age of Transformers","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-01-19 21:44:28","doi":"10.21203/rs.3.rs-3868861/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-03-12T10:14:46+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-03-02T23:57:42+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"80ad1913-6467-4c30-97fa-2e8ef2844339","date":"2024-03-02T21:34:02+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"5290e5a7-9332-455f-8813-4f54a3c974c8","date":"2024-02-23T07:48:13+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-01-27T22:51:36+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-01-27T22:49:38+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2024-01-18T05:34:20+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-01-18T05:28:36+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2024-01-16T05:19:29+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"2a8983d1-9614-4c14-9671-4cf2ba7776fe","owner":[],"postedDate":"January 19th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":28213259,"name":"Physical sciences/Mathematics and computing/Computational science"},{"id":28213260,"name":"Biological sciences/Computational biology and bioinformatics/Machine learning"}],"tags":[],"updatedAt":"2024-05-02T00:53:58+00:00","versionOfRecord":{"articleIdentity":"rs-3868861","link":"https://doi.org/10.1038/s41598-024-60709-z","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2024-05-01 00:53:58","publishedOnDateReadable":"May 1st, 2024"},"versionCreatedAt":"2024-01-19 21:44:28","video":"","vorDoi":"10.1038/s41598-024-60709-z","vorDoiUrl":"https://doi.org/10.1038/s41598-024-60709-z","workflowStages":[]},"version":"v1","identity":"rs-3868861","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3868861","identity":"rs-3868861","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}