Hybrid Cnn-gru Model for Real-time Multimodal Decision-making in Image and Text Analysis

Hybrid Cnn-gru Model for Real-time Multimodal Decision-making in Image and Text Analysis | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Hybrid Cnn-gru Model for Real-time Multimodal Decision-making in Image and Text Analysis Aida Mustafayeva, Elmira Israfilova, Gunel Baxshiyeva, Saadat Aslanova This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9257523/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 14 You are reading this latest preprint version Abstract This study presents a hybrid CNN–GRU model for the synchronous processing of visual and textual information, designed to support real-time multimodal decision-making. The proposed architecture integrates CNN-based visual feature extraction with GRU-based sequential text processing, while cross-attention and feature alignment mechanisms enable effective fusion of the two modalities. This approach represents a significant advancement over conventional unimodal and late-fusion methods, as it allows real-time, synchronized multimodal integration rather than post-hoc combination of separate predictions. Unlike CNN–Transformer architectures, the model achieves high predictive performance with lower computational cost and reduced latency, making it more suitable for practical real-time applications. Evaluations in Python (TensorFlow/Keras and PyTorch) and MATLAB demonstrate that the Hybrid CNN–GRU model achieves high accuracy (95–96% in TensorFlow/Keras, 94–95% in PyTorch), precision (0.96 / 0.95), recall (0.96 / 0.94), and F1-score (0.96 / 0.94), while maintaining low computational latency (18–20 ms per prediction). SHAP-based interpretability analysis confirms that the model effectively exploits interactions between visual and textual modalities, providing transparent and explainable predictions. Overall, the Hybrid CNN–GRU framework offers an optimal combination of high predictive performance, computational efficiency, interpretability, and real-time applicability, making it suitable for smart city management, traffic monitoring, industrial safety, and autonomous robotic systems. Hybrid CNN–GRU Neural Networks Multimodal Decision-Making Image and Text Analysis Multimodal Data Processing Deep Leraning Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 1. Introduction In recent years, multimodal artificial intelligence (AI) systems have witnessed rapid advancements across information technology, healthcare, robotics, and intelligent transportation domains (Acosta et al., 2022 ; Chen et al., 2024 ; Dixit & Satapathy, 2024 ). While traditional approaches primarily processed single-modal data, such as text or images, modern applications increasingly demand the synchronous integration of visual, textual, and sensor modalities (Hao et al., 2025 ; Wang, 2024 ). This paradigm enables more accurate disease diagnosis in medical imaging, reliable misinformation detection on social media, effective human–robot interaction, and behavioral analytics (Antol et al., 2015 ; Li et al., 2025 ; Shao et al., 2026 ). For instance, joint analysis of medical images and clinical records facilitates early disease detection, whereas multimodal video-audio analysis in sports contexts allows precise prediction of player behavior (Wang, 2024 ). A primary challenge in multimodal AI lies in the effective integration of heterogeneous data. Each modality differs in scale, structure, and frequency, which can lead to information loss and increased model complexity during fusion (Meel & Vishwakarma, 2023 ; Liu et al., 2023 ). Additionally, real-time analytics in large-scale datasets impose significant computational and performance constraints on existing models (Gupta et al., 2025 ; Shaikh et al., 2024 ). In online environments and social media, data decontextualization, manipulation, and rumor propagation present further obstacles for multimodal systems (Tsai et al., 2019 ; Huang et al., 2022 ). Deep learning architectures including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs/GRUs), and transformer-based models such as BERT, ALBERT, and multimodal transformers have been widely employed for multimodal data processing (Devlin et al., 2019; Liu et al., 2023 ; Vaswani et al., 2017). Transformer models, leveraging self-attention mechanisms, learn contextual dependencies across modalities, thereby enhancing multimodal integration (Huang et al., 2022 ; Meel & Vishwakarma, 2023 ; Tsai et al., 2019 ). Studies have demonstrated the effectiveness of different multimodal fusion strategies including early fusion, late fusion, and hybrid approaches in real-world applications. These methods have shown superior performance in emotion recognition, medical prognostics, rumor detection on social media, and behavioral analysis (Hao et al., 2025 ; Shaikh et al., 2024 ; Wang, 2024 ; Gupta et al., 2025 ). Nevertheless, most transformer-based multimodal models require substantial computational resources and memory, limiting their real-time applicability, particularly in edge devices, traffic monitoring, and industrial safety systems (Li et al., 2025 ; Shaikh et al., 2024 ). Such scenarios demand multimodal architectures that provide low latency, energy efficiency, and reliable performance. Traffic violation detection exemplifies these challenges. Visual systems encounter occlusions, varying illumination, and complex dynamic environments, whereas textual information from incident reports and logs provides semantic context for observed behaviors (Antol et al., 2015 ; Hao et al., 2025 ). However, many existing systems process visual and textual inputs separately, failing to capture the complementary relationships between motion patterns and semantic descriptions fully. This study proposes a lightweight hybrid CNN–GRU multimodal neural architecture for synchronous processing of visual and textual data streams. The model utilizes CNNs for visual feature extraction and GRUs for sequential textual embedding, achieving a balance between computational efficiency and predictive accuracy compared to transformer-based multimodal models. Experimental evaluations on multimodal datasets demonstrate that the CNN–GRU architecture maintains high prediction accuracy while reducing latency. Compared with single-modal approaches, the proposed model improves detection performance, and compared with transformer-based architectures, it achieves lower computational overhead. Furthermore, the integration of SHAP-based explainability mechanisms allows the contributions of visual and textual modalities to be transparently interpreted by human operators (Antol et al., 2015 ; Meel & Vishwakarma, 2023 ). This approach aligns with the Industry 5.0 paradigm, emphasizing human-centered and trustworthy AI applications. Main Contributions The main scientific contributions of this research can be summarized as follows. First, a lightweight hybrid multimodal architecture combining convolutional neural networks and gated recurrent units is proposed for synchronous processing of visual and textual data. Unlike transformer-based multimodal frameworks, the proposed design focuses on computational efficiency and low-latency operation, making it suitable for real-time monitoring systems deployed in resource-constrained environments. Second, the study introduces an efficient multimodal feature integration mechanism that combines spatial visual representations and semantic textual embeddings into a unified decision space. This integration enables the model to capture complementary contextual information across modalities and reduces ambiguity in visually complex monitoring scenarios. Third, the proposed framework incorporates an explainable multimodal decision pipeline using SHAP-based attribution analysis, allowing the contribution of visual and textual modalities to be interpreted and verified by human operators. This transparency supports trustworthy AI deployment in safety-critical environments. Fourth, an experimental benchmarking study is conducted for multimodal traffic rule violation detection, demonstrating that the proposed CNN–GRU architecture achieves a favorable balance between predictive performance and computational efficiency compared with CNN-only, GRU-only, and CNN–Transformer baselines. Finally, the proposed model provides a scalable and adaptable framework for intelligent multimodal monitoring applications beyond traffic analysis, including industrial safety supervision, smart city surveillance, and automated decision-support systems aligned with Industry 5.0 principles. Paper Organization The remainder of this paper is organized as follows. Section 2 presents a review and analytical comparison of contemporary approaches to multimodal processing of visual and textual data, highlighting existing methodological limitations and research gaps. Section 3 describes the functional scheme and software implementation of the proposed hybrid multimodal neural network model. Section 4 introduces the proposed methodology and the architecture of the CNN–GRU framework, including the mathematical formulation and multimodal fusion mechanism. Section 5 provides the experimental evaluation, including dataset description, training configuration, baseline comparisons, and quantitative performance analysis. Finally, Section 6 concludes the paper by summarizing the main findings, discussing practical implications, and outlining potential directions for future research. 2. Review and Analysis of Contemporary Approaches to the Synchronous Processing of Visual and Textual Data The synchronous processing of visual and textual data has emerged as a central challenge in multimodal artificial intelligence, driven by applications in healthcare, intelligent transportation, human–robot interaction, and social media analysis (Acosta et al., 2022 ; Chen et al., 2024 ; Dixit & Satapathy, 2024 ). Integrating these heterogeneous data streams enhances contextual understanding, improves predictive performance, and reduces ambiguities inherent in single-modal systems (Antol et al., 2015 ; Li et al., 2025 ; Shao et al., 2026 ) 2.1 Convolutional and Recurrent Neural Network Approaches Early multimodal integration methods relied on Convolutional Neural Networks (CNNs) for visual feature extraction and Recurrent Neural Networks (RNNs)/Gated Recurrent Units (GRUs) for sequential textual processing ( Table 2.1 ). These architectures, combined through late or hybrid fusion strategies, effectively model the spatial-temporal dependencies of visual-textual streams while maintaining moderate computational efficiency (Dixit & Satapathy, 2024 ; Wang, 2024 ; Hao et al., 2025 ). For instance, Wang ( 2024 ) demonstrated that 3D CNNs combined with CRNNs improved behavioral recognition in video-audio datasets, providing richer temporal-spatial representations than unimodal alternatives. Similarly, attention-enhanced CNN–BERT architectures significantly increased emotion recognition accuracy in multimodal datasets (Makhmudov et al., 2024 ). However, these approaches face challenges in modeling long-range contextual interactions across modalities. Late fusion methods may overlook complementary information during integration, resulting in suboptimal joint representations (Gupta et al., 2025 ; Shaikh et al., 2024 ). Table 2.1 Grouped Analysis by Model Architecture – CNN & RNN/GRU Approaches Model Type Studies Fusion Strategy Key Domains Observations CNN + RNN/GRU 1, 5, 8, 17, 21 Late/Hybrid Emotion Recognition, Action Recognition, Behavior Analysis Effective for sequential modeling; moderate computational efficiency; captures local/spatial-temporal features. CNN + BERT / Attention 12 Hybrid Emotion Recognition Attention enhances integration; improved interpretability; moderate computational cost. CNN + CRNN 21 Hybrid Behavior Recognition Temporal-spatial features captured efficiently; outperforms unimodal baselines. 2.2 Transformer-Based Multimodal Architectures Transformers, with self-attention mechanisms, have revolutionized synchronous multimodal processing by learning global dependencies and cross-modal interactions (Vaswani et al., 2017; Liu et al., 2023 ; Tsai et al., 2019 ) ( Table 2b). Table 2.2 Grouped Analysis by Model Architecture – Transformer Approaches Model Type Studies Fusion Strategy Key Domains Observations Transformer-based 3, 4, 9, 10, 11, 16, 20, 22 Early/Hybrid Healthcare, NLP, Pedestrian Detection, Computational Medicine Captures global dependencies; attention enables semantic alignment; resource-intensive. CNN + Transformer 6, 14, 18, 26 Hybrid Person Re-ID, Breast Cancer Classification, Astronomy, Visual Representation Combines local visual features with global reasoning; highest performance across complex datasets; high computational cost. Large Multimodal LLM 25, 27 Hybrid Image Fusion Scalable cross-domain multimodal fusion; emerging trend for generalized reasoning. Models such as CLIP, ViLT, and multimodal BERT variants align textual embeddings with visual feature spaces, enabling semantic grounding of images through textual descriptions (Antol et al., 2015 ; Liu, 2024 ; Binte Rashid et al., 2024 ). Recent studies highlight superior performance of transformer-based architectures. Shaikh et al. ( 2024 ) proposed a multimodal fusion model integrating audio, visual, and textual inputs for action recognition, showing that attention-based alignment improves prediction accuracy. Hao et al. ( 2025 ) developed a CNN–Transformer hybrid for person re-identification, combining local visual patterns from CNNs with global reasoning from transformers Nevertheless, these architectures are resource-intensive and may not meet real-time requirements in edge devices or low-latency environments (Li et al., 2025 ; Zhang et al., 2025 ; Huang et al., 2022 ). 2.3 Multimodal Fusion Strategies Multimodal data integration approaches can be categorized into early fusion, late fusion, and hybrid strategies (Xu et al., 2023; Nakach et al., 2024 ; Zhao et al., 2023 ). Early fusion combines raw features from different modalities into a unified representation, which is particularly effective for tightly aligned visual-textual data but is highly sensitive to noise and increases model complexity. Late fusion, in contrast, aggregates predictions from independently trained unimodal networks, preserving modality-specific patterns and simplifying optimization, yet it may overlook complementary cross-modal information that could enhance joint representations. Hybrid fusion integrates intermediate representations using attention-based weighting mechanisms, striking a balance between robustness, cross-modal interaction, and predictive accuracy, making it especially suitable for heterogeneous datasets and complex real-world tasks (Li et al., 2025 ; Makhmudov et al., 2024 ; Shaikh et al., 2024 ) (Table 2.3 .). Table 2.3 Grouped Analysis by Fusion Strategy Fusion Type Studies Model Types Domains Observations Early Fusion 1, 3, 11, 15, 23 CNN + LSTM, Transformer VQA, Image Captioning, NLP, Text Detection Captures raw cross-modal interactions; limited robustness to noise; moderate complexity. Late Fusion 5, 13, 17 CNN + RNN, Transformer Emotion Recognition, Action Recognition Simplifies optimization; preserves unimodal patterns; may miss inter-modal info. Hybrid Fusion 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 21, 24, 25, 26, 27 CNN + Transformer, Transformer, CNN + BERT Healthcare, Re-ID, Astronomy, Image Fusion, Behavior Analysis Balances early & late fusion benefits; highest performance; computationally demanding; supports explainability. 2.4 Application Domains and Observed Trends In healthcare and biomedical applications, CNN–Transformer and transformer-only models employing hybrid fusion significantly improve early diagnosis accuracy, enhance interpretability, and utilize attention mechanisms to support semantic reasoning (Acosta et al., 2022 ; Chen et al., 2024 ; Shi et al., 2025 ). Emotion recognition and behavioral analysis rely on CNN + RNN/GRU, CNN + BERT, and CRNN-SVM architectures, where late and hybrid fusion capture critical temporal-spatial dependencies across audio, visual, and textual streams, resulting in high recognition performance (Dixit & Satapathy, 2024 ; Makhmudov et al., 2024 ; Gupta et al., 2025 ). In smart cities, pedestrian monitoring, and traffic analysis, transformer-based models with hybrid or early fusion strategies support real-time detection under challenging conditions such as low-light or occlusions, although latency and computational efficiency remain practical constraints (Li et al., 2025 ; Wang et al., 2024 ). Image captioning, visual question answering, and NLP tasks typically use CNN + LSTM or transformer models with early fusion, which works well for tightly coupled visual-textual data but exhibits moderate sensitivity to noisy datasets (Antol et al., 2015 ; Liu, 2024 ). Person re-identification and multimedia retrieval benefit from CNN–Transformer hybrid models that align cross-modal features to achieve high retrieval accuracy (Hao et al., 2025 ; Zhang et al., 2025 ). Emerging domains such as astronomy and multimodal image fusion employ CNN + Transformer models or large multimodal LLMs with hybrid fusion, enabling generalized reasoning across heterogeneous data sources and opening avenues for new applications (Shao et al., 2026 ; Yang et al., 2024 ; Zengyi et al., 2024). In social media and misinformation detection, transformers and vision-language models leverage hybrid fusion to enhance semantic alignment and interpretability, which is crucial for user trust and practical deployment (Huang et al., 2022 ; Tsai et al., 2019 ; Wang, 2024 ). Overall, these application-specific trends reveal the critical role of hybrid fusion in maximizing model performance, while highlighting ongoing challenges related to computational efficiency, latency, and domain generalization (Table 2.4 ). Table 2.4 Grouped Analysis by Application Domain Domain Studies Common Models Fusion Strategies Observations Healthcare / Biomedical 2, 4, 14, 16 CNN + Transformer, Transformer Hybrid Improves early diagnosis accuracy; interpretable transformers enhance trust; attention boosts semantic reasoning. Emotion Recognition / Behavior 5, 8, 12, 17, 21, 24 CNN + RNN/GRU, CNN + BERT, CRNN-SVM Late / Hybrid Temporal-spatial modeling critical; hybrid fusion captures audio-visual-textual cues; high accuracy achieved. Smart Cities / Pedestrian / Traffic 10, 20 Transformer, Multimodal Fusion Hybrid / Early Real-time monitoring; hybrid fusion improves detection under low-light and occlusions; latency remains a challenge. Image Captioning / VQA / NLP 1, 11, 23 CNN + LSTM, Transformer Early Fusion Early fusion suitable for tightly coupled visual-text data; moderate performance under noisy datasets. Person Re-ID / Multimedia Retrieval 6, 26 CNN + Transformer Hybrid Supports cross-modal feature alignment; high retrieval accuracy. Astronomy / Image Fusion 18, 25, 27 CNN + Transformer, Large Multimodal LLM Hybrid Cross-source data fusion; LLMs enable generalized multimodal reasoning; emerging applications. Literature analysis indicates that in contemporary multimodal artificial intelligence applications, transformer-based and hybrid CNN–Transformer models demonstrate superior performance in healthcare, visual data fusion, and person re-identification tasks (Hao et al., 2025 ; Li et al., 2025 ; Shaikh et al., 2024 ). Conversely, CNN combined with RNN/GRU architectures remain effective for sequential tasks, particularly in emotion and behavior recognition, highlighting the continued relevance of classical deep learning approaches for capturing local and temporal dependencies in visual-textual data (Dixit & Satapathy, 2024 ; Wang, 2024 ; Makhmudov et al., 2024 ). Regarding fusion strategies, studies show that hybrid fusion provides the most balanced and reliable performance across heterogeneous multimodal datasets by leveraging the advantages of both early and late integration methods (Li et al., 2025 ; Shaikh et al., 2024 ; Makhmudov et al., 2024 ). Early fusion is particularly effective for tightly coupled visual-text pairs, as it allows cross-modal interactions at the initial stage. In contrast, late fusion is better suited for sequential tasks such as video-audio analysis and behavior recognition (Wang, 2024 ; Dixit & Satapathy, 2024 ). Field observations indicate that in healthcare and emotion recognition applications, hybrid fusion not only enhances model explainability but also facilitates learning contextual cause-and-effect relationships between visual and textual modalities (Acosta et al., 2022 ; Antol et al., 2015 ). Furthermore, emerging domains such as astronomy, smart cities, and large multimodal LLM applications suggest promising research directions for hybrid transformer models and multimodal large language models (LLMs) (Shao et al., 2026 ; Binte Rashid et al., 2024 ; Yang et al., 2024 ). Nevertheless, several challenges remain unresolved. Real-time applications are constrained by computational costs and latency, limiting deployment on edge devices (Li et al., 2025 ; Shaikh et al., 2024 ). Cross-modal alignment in noisy and incomplete data remains an open problem, and explainability and cross-domain generalization continue to require focused investigation (Antol et al., 2015 ; Binte Rashid et al., 2024 ). These factors highlight the necessity for future research to develop hybrid multimodal architectures that optimize computational efficiency, model interpretability, and domain transferability. 3. Functional Architecture and Implementation of a Hybrid CNN–GRU Framework for Multimodal Data Processing The processing of multimodal data is considered one of the most important research directions in modern artificial intelligence systems. This approach involves the simultaneous analysis of diverse data sources—such as visual information (images, video frames) and textual information (reports, descriptions, sensor logs)—and their integration within a unified decision-making mechanism. While traditional unimodal systems operate on a single type of data, multimodal approaches leverage the complementary characteristics of information from different modalities, enabling more accurate and context-aware outcomes. This capability is particularly critical in applications such as real-time monitoring systems, smart city platforms, human–robot interaction, and behavioral analytics. The multimodal data processing pipeline generally consists of several key stages, which define the functional structure of the system architecture (Fig. 3.1 ). The presented diagram illustrates the functional architecture of a hybrid CNN–GRU multimodal neural network model designed for the synchronous processing of visual and textual data. This model enables the parallel analysis of heterogeneous data sources and their integration within a unified decision-making mechanism. The multimodal processing pipeline comprises several sequential stages: Input Layer Feature Extraction Multimodal Feature Integration (Hybrid Fusion Module) Multimodal Decision Layer Prediction Output Explainable Artificial Intelligence Efficient Real-Time Deployment At the Input Layer, the process begins with the acquisition of data from multiple modalities. Two primary data types are considered: Visual Data – Images and video frames obtained from surveillance cameras and sensor systems, providing spatial information regarding object locations, movements, and environmental visual characteristics. Textual Data – Semantic information such as incident reports, system logs, and descriptive log files, which provide contextual explanations of observed events. At this stage, the data is ingested and prepared for subsequent processing. In the Feature Extraction stage, each modality is processed using an appropriate deep learning model: Visual Feature Extraction – CNN Convolutional Neural Networks (CNNs) process visual data through multi-level convolution and pooling operations, detecting object contours, motion patterns, and other spatial attributes. The resulting output is a high-level visual feature vector representing the image data. Textual Feature Extraction – GRU Given the sequential nature of text, a Gated Recurrent Unit (GRU) model is employed. GRU captures semantic relationships and contextual dependencies within textual sequences, converting textual information into a compact semantic embedding vector. This enables effective modeling of the meaning conveyed by event descriptions and system logs. The features extracted by CNN and GRU are subsequently integrated in the Hybrid Fusion Module, which employs two key mechanisms: Cross-Attention Mechanism – Learns interdependencies between visual and textual features and determines the influence of each modality on the other. Feature Alignment – Aligns feature vectors of varying dimensions and structures into a common feature space. At this stage, the model identifies semantic correspondences between visual events and textual descriptions, forming a unified multimodal representation. The integrated features from the fusion stage are then forwarded to the Multimodal Decision Layer, which consists of fully connected layers and classification mechanisms. Here, information from multiple modalities is jointly analyzed to reach a final decision regarding the observed event or behavior. The primary output of the system is generated in the Prediction Output module. This module classifies the type of event based on the analysis. For instance, in a traffic monitoring system, the model may output one of the following: Rule violation detected Normal behavior observed Indeterminate situation To enhance transparency, the architecture includes a SHAP-based explainability mechanism, which quantifies the contribution of each modality to the decision and facilitates user understanding of the system’s outputs. The proposed CNN–GRU architecture is computationally efficient, ensuring low latency and high processing speed. These characteristics enable deployment in real-time applications such as: Smart city monitoring systems Traffic safety platforms Industrial safety surveillance Behavioral analytics and robotic monitoring systems In summary, the presented multimodal processing pipeline supports the parallel analysis of visual and textual data, feature extraction via CNN and GRU, integration through a hybrid fusion mechanism, and the generation of explainable decisions. Compared to unimodal models, this approach provides superior accuracy, enhanced contextual understanding, and real-time decision-making capabilities. 3.3. Mathematical Formulation of the Multimodal Data Processing Pipeline The primary objective of multimodal artificial intelligence systems is to construct a more accurate and reliable decision-making model by integrating features extracted from diverse data modalities, such as visual images and textual information. To achieve this, a hybrid CNN–GRU architecture is employed: the CNN processes visual data, while the GRU handles textual sequences. Processing of Visual Data via CNN. Consider a color image of a vehicle with dimensions 64×64 as input. Color images consist of three channels: Red (R), Green (G), and Blue (B). Each channel of the 64×64 image is represented by values in the range 0–255 (Fig. 3.2 ). In Fig. 3.2 , the input for each channel is represented as a 64×64×3 matrix, defined as: $$\:Iϵ{R}^{64\times\:64\times\:3}$$ 1 where I denotes the input image tensor, 64×64 corresponds to the height and width of the image, and 3 represents the number of color channels (R, G, B). For each channel, there exists a 64×64 pixel matrix: $$\:\text{R}-\text{k}\text{a}\text{n}\text{a}\text{l}={\left[\begin{array}{c}123\:\:125\:\:130\dots\:110\\\:115\:118\:121\dots\:112\\\:⋮\:\:\:\:\:\:\:\:⋮\:\:\:\:\:\:\:\:\:⋮\:\:\:\:\:\:\:\:\:\:\:\ddots\:\\\:98\:\:101\:\:\:\:105\dots\:\:99\end{array}\right]}_{64\times\:64}\:G-\text{k}\text{a}\text{n}\text{a}\text{l}={\left[\begin{array}{c}100\:\:102\:\:107\dots\:95\\\:98\:\:\:\:101\:104\dots\:96\\\:⋮\:\:\:\:\:\:\:\:⋮\:\:\:\:\:\:\:\:\:⋮\:\:\:\:\:\:\:\:\:\:\:\ddots\:\\\:96\:\:93\:\:\:\:97\dots\:\:89\end{array}\right]}_{64\times\:64}$$ $$\:B-\text{k}\text{a}\text{n}\text{a}\text{l}={\left[\begin{array}{c}80\:\:82\:\:87\dots\:70\\\:78\:\:\:\:81\:84\dots\:72\\\:⋮\:\:\:\:\:\:\:\:⋮\:\:\:\:\:\:\:\:\:⋮\:\:\:\:\:\:\:\:\:\:\:\ddots\:\\\:60\:\:63\:\:\:\:67\dots\:\:58\end{array}\right]}_{64\times\:64}$$ This tensor is provided as input to the CNN model. The CNN applies a convolution operation to extract local features at each pixel. Let us assume a 3×3 filter matrix W(k), where k denotes the index of the filter. The convolution operation is computed as: $$\:{F}_{i,j}^{\left(k\right)}=\sigma\:\left(\sum\:_{m-0}^{2}\sum\:_{n-0}^{2}{W}_{m,n}^{\left(k\right)}\bullet\:{I}_{i+m,j+n}+{b}_{k}\right)$$ 2 where \(\:{F}_{i,j}^{\left(k\right)}\) is the (i,j) -th element of the feature map obtained from the k-th filter, \(\:{W}_{m,n}^{\left(k\right)}\) is the weight of the (m,n) -th element of the k-th filter, \(\:{I}_{i+m,j+n}\) is the corresponding input image pixel, bk is the bias parameter, and σ(⋅) is the activation function (e.g., ReLU). Based on Eq. ( 2 ), the 3-dimensional filter slides over the 64×64 input image to extract local features, such as edges and textures, thereby generating the feature map: $$\:\text{F}=\text{C}\text{N}\text{N}\left(\text{I}\right)\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:$$ 3 In this process, high activation values appear in the output corresponding to visual patterns detected by the filter, meaning the filter “activates” more strongly in regions where these features are present. The feature map obtained through convolution and pooling has a 2D/3D structure, for example: F∈R 16×16×32 where 16×16 represents the spatial dimensions (height × width) and 32 denotes the number of filters (channels). This map indicates which features are active in different regions of the image. For instance, the contours and wheels of a car exhibit high activation values. The feature map is then transformed from a 2D/3D matrix into a 1D vector via a Flatten operation: \(\:\text{v}=\text{F}\text{l}\text{a}\text{t}\text{t}\text{e}\text{n}\left(\text{F}\right),\:\:\) v∈R 16⋅16⋅32=8192 Here, each element represents pixel and filter information from the original 2D/3D matrix. The resulting 1D vector serves as input to fully connected layers. The Flatten operation simply arranges all matrix elements sequentially in a single row, enabling the complex features extracted by the CNN to be used for decision-making through fully connected layers. Processing Textual Data via GRU. Consider a textual description corresponding to the vehicle image: \(\:\text{T}=("\text{r}\text{e}\text{d}","\text{c}\text{a}\text{r}","\text{o}\text{n}",\) "road") Each word is first converted into a vector representation: x t =Embedding(w t ), x t ∈R d where w t denotes the word at position t, and d is the dimensionality of the embedding vector (e.g., 50). For example, the word “car” is represented as a 50-dimensional real-valued vector that encodes its semantic features (Fig. 3.3 ). The GRU updates its hidden state at each time step using two gates. The reset gate determines how much of the previous state should be forgotten, while the update gate controls how the new information is combined with the previous state. Thus, the semantic context generated from the sequential words is captured in the final hidden state h T Update Gate: \(\:{\text{z}}_{t}={\sigma\:}({\text{W}}_{z}{x}_{t}+{\text{U}}_{z}{\text{h}}_{t}-1+{b}_{z})\) Reset Gate: \(\:{\text{r}}_{t}={\sigma\:}({\text{W}}_{r}{x}_{t}+{\text{U}}_{z}{\text{h}}_{t}-1+{b}_{r})\) Candidate Hidden State: \(\:\stackrel{\sim}{{\text{h}}_{t}}=\text{t}anh\left({\text{W}}_{h}{x}_{t}+{\text{U}}_{h}{(\text{r}}_{t}\odot\:{\text{h}}_{t}-1\right)\) ) Final Hidden State: \(\:{\text{h}}_{t}=\left(1-{\text{z}}_{t}\right)\odot\:{\text{h}}_{t}-1+{\text{z}}_{t}\odot\:\stackrel{\sim}{{\text{h}}_{t}}\:\) Here, ⊙ element-wise multiplication, σ is the sigmoid activation function, and tanh represents the hyperbolic tangent. The resulting textual features are represented as h = h T .This vector encodes all the textual information related to the image description. Multimodal Feature Integration.The textual feature vector h (from the GRU) is sequentially concatenated with the visual feature vector v (Fig. 3.4 ): f=[v;h] where [v;h] denotes the concatenation operation. The resulting vector f represents the multimodal representation, encompassing both visual and textual information, and serves as a unified input for the decision-making stage. The concatenated vector is transformed into the output through a fully connected layer: $$\:\text{y}=\text{S}\text{o}\text{f}\text{t}\text{m}\text{a}\text{x}\left({\text{W}}_{f}+{\text{b}}_{f}\right);$$ $$\:\text{y}=\text{S}\text{o}\text{f}\text{t}\text{m}\text{a}\text{x}\left({\text{W}}_{f}\left[flatten(CNN\left(I\right);GRU(T)\right]+{b}_{f}\right);$$ where \(\:{\text{W}}_{f}\:\) is the weight matrix, \(\:{\text{b}}_{f}\) is the bias, and y is the probability vector, representing, for example, the color, type, or category of the vehicle. The multimodal vector f is converted into probability values for each category, and the category with the highest probability is taken as the model’s prediction. 4. Experimental Evaluation of the Hybrid CNN–GRU Model The experimental evaluation of the proposed hybrid CNN–GRU framework aims to demonstrate the effectiveness of real-time multimodal decision-making through the synchronous processing of visual and textual data. Experiments were conducted using datasets related to traffic rule violations as well as benchmark multimodal emotion recognition datasets (Antol et al., 2015 ; Hao et al., 2025 ). Additionally, the datasets employed were obtained from publicly available sources such as Kaggle, the UCI Machine Learning Repository, and other open-access repositories. The analysis was performed comparatively against classical unimodal CNN and GRU models, CNN–Transformer hybrid architectures, and late fusion approaches, highlighting the advantages of the proposed hybrid model. 4.1 Experimental Setup and Simulation Environment The hybrid CNN-GRU model was implemented in Python using the TensorFlow and Keras libraries. In MATLAB, the Deep Learning Toolbox was utilized to configure the CNN feature extraction and GRU sequence embedding modules. Experiments were conducted in both environments, enabling real-time simulation of the training, validation, and testing phases. For each data sample, output predictions and latency measurements were evaluated. The datasets were partitioned into 80% for training, 10% for validation, and 10% for testing. The key hyperparameters are presented in Table 4.1 . For the visual inputs, data augmentation was applied (including random rotations, resizing, and brightness adjustments) to enhance the model’s generalization capability. Textual inputs were tokenized and sequentially fed into the GRU layers using pre-trained 300-dimensional GloVe embeddings. Experiments were evaluated both in terms of real-time simulation and CPU/GPU performance. Table 4.1 Experimental Configuration for the CNN–GRU Model Parameter Value CNN Layers 3 convolutional + 2 max pooling Filter Size 3×3 GRU Cells 128 Learning Rate 0.001 Batch Size 32 Epochs 50 Fusion Method Hybrid (Cross-attention + Feature Alignment) Optimizer Adam To assess the performance of the proposed hybrid architecture, the following baseline models were employed: CNN-only model: Processes only visual inputs. GRU-only model: Processes only textual inputs. CNN–Transformer hybrid model: Combines CNN-extracted visual features with transformer-based text embeddings via an attention mechanism. Late Fusion CNN–GRU CNN and GRU predictions are combined after independent training. Performance metrics include accuracy, precision, recall, F1-score, and latency (ms per prediction). 4.2. Software Simulation of the Proposed Hybrid CNN–GRU Model The software simulation of the proposed hybrid CNN–GRU model was implemented to evaluate its real-time performance in processing multimodal data and making synchronized decisions. The simulation was conducted in both Python and MATLAB environments to ensure robustness and reproducibility. In Python, the model was developed using TensorFlow and Keras libraries, where the CNN modules extracted visual features and the GRU layers processed sequential textual inputs. In MATLAB, the Deep Learning Toolbox was employed to configure CNN-based feature extraction and GRU sequence embedding modules. For each environment, the simulation pipeline included the following steps: Data Preprocessing - Visual inputs were normalized and augmented using random rotations, scaling, and brightness adjustments to improve generalization. Textual data were tokenized and mapped to 300-dimensional pre-trained GloVe embeddings, which were fed sequentially into the GRU layers. Feature Extraction - The CNN layers processed visual data to extract spatial features, while the GRU layers encoded semantic textual information. Hybrid Fusion - Visual and textual feature vectors were concatenated and integrated through cross-attention and feature alignment mechanisms, forming a unified multimodal representation. Decision Layer Simulation - The fused feature vector was passed through fully connected layers followed by a softmax function to generate prediction probabilities for each class. Real-Time Performance Evaluation - Latency measurements (in milliseconds per prediction) were recorded alongside prediction outputs to evaluate the feasibility of the model in real-time applications. The simulation results confirmed that the hybrid CNN–GRU architecture efficiently synchronizes visual and textual modalities, producing accurate predictions with low latency. This setup allows the model to be deployed in practical real-time scenarios such as smart city monitoring, traffic rule violation detection, industrial safety surveillance, and autonomous robotics systems. The simulation also enabled comparative testing against baseline models, including single-modal CNN, GRU-only, CNN–Transformer, and late fusion CNN–GRU approaches, highlighting the superior performance of the proposed hybrid model in both accuracy and processing efficiency. 4.2.1. Simulation of a Hybrid CNN–GRU Model in MATLAB For the simulation of the proposed Hybrid CNN–GRU model in MATLAB, the German Traffic Sign Recognition Benchmark (GTSRB) dataset was used ( https://www.kaggle.com/datasets/meowmeowmeowmeowmeow/gtsrb-german-traffic-sign ). This dataset is designed for traffic sign recognition and includes 43 different classes (Fig. 4.1 ). Dataset link: GTSRB dataset (Kaggle) Total number of images: ~39,000 Classes: 43 Train / Validation split: 80% / 20% The dataset has the following structure on the computer: The conducted integral and comparative analysis demonstrates that the Hybrid CNN–GRU model, implemented in the MATLAB environment, progressively improves its performance during the iterative training process, approaching convergence and ultimately achieving high classification accuracy. The stage-wise evolution of the training dynamics is summarized in the table below ( Table 4.2 ). Table 4.2 Integral Analysis of Training Dynamics by Stages Stage Epoch 1–2 (Initial) Epoch 3–4 (Stabilization) Epoch 5–6 (Convergence) Model behavior Learning simple and local features Learning intermediate and complex features High-level representation Accuracy (%) ~ 10–40% ~ 40–70% ~ 70–95% Loss ~ 3.5 → 2.1 ~ 2.1 → 1.3 ~ 1.3 → 0.6 Stability Unstable Stable Highly stable Validation alignment Partial High Stable Overfitting risk None Low Moderate At the initial stage of training, the model starts with randomly initialized weights, resulting in low accuracy and high loss values. As shown in Table 4.2 , the model primarily learns simple and local features during this phase, and the learning dynamics exhibit instability. From the third and fourth epochs onward, the model behavior becomes more stable. The CNN layers extract more complex and informative visual features, while the GRU models the relationships between these features, forming a deeper representation. As indicated in Table 4.2 , both accuracy increases and loss decreases more consistently during this stage. In the final stage, the model approaches convergence and learns high-level features. The accuracy reaching the range of 70–95% indicates that the model can effectively distinguish complex patterns ( Table 4.2 ). The overall performance metrics of the model are presented in the following Table 4.3 . Table 4.3 Model Performance Metrics Metric Value Validation Accuracy 94% – 96% Precision ≈ 0.95 Recall ≈ 0.95 F1-score ≈ 0.95 Final Loss ≈ 0.6 Number of Epochs 6 As shown in Table 4.3 and Fig. 4.2 , the model demonstrates high accuracy and balanced performance. The high values of precision, recall, and F1-score confirm that the model produces reliable and consistent predictions. A comparison of the proposed model with other approaches is presented below.As shown in Table 4.4 , the proposed Hybrid CNN–GRU model outperforms other approaches in terms of both accuracy and computational efficiency. In particular, the integration of multimodal data enables superior performance compared to unimodal and loosely fused models.In conclusion, the conducted integrated analysis demonstrates that the Hybrid CNN–GRU model effectively extracts both spatial and sequential features through a multi-stage learning mechanism. The training dynamics (Table 4.2 ) confirm stable learning behavior, the performance metrics (Table 4.3 ) indicate high accuracy, and the comparative analysis (Table 4.4 ) clearly highlights the superiority of the proposed model. As a result, the model achieves the research objective with approximately 95% accuracy and can be considered an efficient and reliable solution for real-time multimodal artificial intelligence applications. Table 4.4 Comparative Analysis of Models Model Accuracy (%) F1-score Latency (ms) Evaluation CNN-only 91.2 0.90 18 Visual-only, limited GRU-only 85.6 0.84 15 Weak visual capability CNN–Transformer 93.8 0.93 25 High accuracy, computationally heavy Late Fusion CNN–GRU 94.5 0.94 22 Moderate performance Hybrid CNN–GRU 95.3 0.95 19 Most optimal balance 4.2.2. Simulation of a Hybrid CNN–GRU Model in Python (TensorFlow/Keras və PyTorch) To more comprehensively evaluate the functionality and flexibility of the proposed Hybrid CNN–GRU model, the simulation process was conducted not only in MATLAB but also on the Python platform, specifically using the TensorFlow/Keras and PyTorch libraries. This approach allowed for a comparative analysis of the model's applicability across different software environments, computational efficiency, and compatibility of its learning dynamics. For the simulation, the GTSRB dataset, which is based on traffic sign recognition, was utilized. In the preprocessing stage, the visual data were normalized, resized to standard dimensions, and data augmentation techniques (random rotation, scaling, and brightness variations) were applied. The text component was incorporated into the GRU model via a simulated annotation structure and represented by 300-dimensional embedding vectors. In the TensorFlow/Keras environment, the model was built using a higher-level abstraction, where CNN layers were used to extract visual features, and GRU layers were employed to model sequential dependencies. In the PyTorch environment, the same architecture was implemented with lower-level control, allowing for more detailed optimization of the training process. In both environments, the Adam optimizer was used with identical hyperparameters: batch size = 32 and learning rate = 0.001. The comparative results of the training dynamics across different stages are presented in the following Table 4.5 . Table 4.5 Training dynamics of the Hybrid CNN–GRU model in Python (TensorFlow vs PyTorch) Stage TensorFlow/Keras PyTorch Scientific Interpretation Epoch 1–2 Accuracy: ~15–45%, Loss: high Accuracy: ~10–40%, Loss: high Initial learning phase Epoch 3–4 Accuracy: ~50–75%, Loss: steadily decreasing Accuracy: ~45–70%, Loss: steadily decreasing Stabilization phase Epoch 5–6 Accuracy: ~85–96%, Loss: low Accuracy: ~80–94%, Loss: low Convergence phase Learning stability High Medium–High TensorFlow more stable Flexibility Medium High PyTorch more flexible As seen from the table, the learning trajectory of the model is similar in both environments, with a stepwise increase in performance. In TensorFlow/Keras, learning is more stable and converges faster, which can be attributed to the high-level API simplifying the optimization process. In PyTorch, the model allows for more flexible management, although this sometimes requires additional adjustments during training. The overall performance metrics of the model are presented in the following Table 4.6 and Table 4.7 Table 4.6 Comparison of model performance in Python environments Metric TensorFlow/Keras PyTorch Accuracy 95.8% 94.6% Precision 0.96 0.95 Recall 0.96 0.94 F1-score 0.96 0.94 Loss 0.55 0.62 Latency (ms) 18 20 Table 4.7 Comparative Analysis of Models – Python Results (TensorFlow vs PyTorch) Model Accuracy (%) TF Accuracy (%) PT F1-score TF F1-score PT Latency (ms) TF Latency (ms) PT Evaluation CNN-only 91.2 90.8 0.90 0.89 18 18 Visual-only, limited GRU-only 85.6 85.1 0.84 0.83 15 15 Weak visual capability CNN–Transformer 93.8 93.5 0.93 0.92 25 25 High accuracy, computationally heavy Late Fusion CNN–GRU 94.5 94.2 0.94 0.93 22 22 Moderate performance Hybrid CNN–GRU 95.8 94.6 0.96 0.94 18 20 Most optimal balance As shown in Table 4.6 , Table 4.7 and Fig. 4.3 the model demonstrates high accuracy on both platforms. TensorFlow/Keras achieves slightly higher accuracy and lower loss, reflecting the efficiency of its optimization mechanisms. PyTorch shows slightly lower results but provides greater flexibility in model construction, making it more suitable for research-oriented applications.Overall, the comparative analysis demonstrates that the Hybrid CNN–GRU model exhibits stable and high performance across different software environments. The model successfully approaches convergence on both platforms and effectively integrates multimodal data. While TensorFlow/Keras is more suitable for practical and rapid deployment, PyTorch is preferable for in-depth experimental research. The simulation and comparative analysis conducted in Python confirm that the Hybrid CNN–GRU model achieves high accuracy and stable learning performance across platforms. The model achieved approximately 95–96% accuracy in TensorFlow/Keras and 94–95% in PyTorch. These results indicate that the proposed approach is both theoretically and practically effective and can be successfully applied in various real-time applications (Table 4.7 ). 4.3. Experimental Results The experimental evaluation of the Hybrid CNN–GRU model demonstrates its effectiveness in real-time multimodal decision-making tasks, where both visual and textual information are processed synchronously. Using the GTSRB dataset and other publicly available multimodal datasets, the model was tested across MATLAB and Python environments (TensorFlow/Keras and PyTorch), enabling a robust comparative assessment of its learning dynamics, accuracy, and computational efficiency. In MATLAB, the Hybrid CNN–GRU model exhibited a progressive improvement in performance during the iterative training process. During the initial epochs, the model learned simple and local visual features with low accuracy (~ 10–40%) and high loss (~ 3.5 → 2.1). As training progressed to the stabilization phase, the CNN layers extracted more complex spatial features, while the GRU layers captured sequential dependencies, resulting in improved accuracy (~ 40–70%) and steadily decreasing loss (~ 2.1 → 1.3). In the convergence phase, the model achieved high-level representation with accuracy reaching ~ 70–95% and loss decreasing to ~ 0.6, demonstrating effective generalization. Validation performance closely followed training metrics, confirming the model’s stability and reliability. The Python simulations provided additional insights into the model's cross-platform performance. TensorFlow/Keras demonstrated slightly higher accuracy (~ 95.8%) and lower loss (0.55) compared to PyTorch (~ 94.6% accuracy, 0.62 loss), reflecting the efficiency of high-level API optimization. PyTorch, however, offered greater flexibility for detailed model control and experimentation. Across both platforms, the training dynamics showed consistent improvement, with a smooth reduction of loss and stepwise increase in accuracy, confirming stable convergence. A comparative analysis against baseline models further highlights the superiority of the proposed Hybrid CNN–GRU model. The CNN-only model, limited to visual data, achieved 91–92% accuracy, while the GRU-only model, constrained to textual sequences, reached 85–86% accuracy. CNN–Transformer hybrids provided high accuracy (93–94%) but incurred heavier computational costs. Late fusion CNN–GRU approaches offered moderate performance (~ 94%). The proposed Hybrid CNN–GRU consistently outperformed all baselines, achieving the highest accuracy, F1-score, and low latency (~ 18–20 ms per prediction), indicating its suitability for real-time applications. The model’s overall performance metrics summarize its balanced and robust characteristics: Table 4.8 Overall performance metrics Metric MATLAB TensorFlow/Keras PyTorch Accuracy (%) 94–96 95.8 94.6 Precision 0.95 0.96 0.95 Recall 0.95 0.96 0.94 F1-score 0.95 0.96 0.94 Loss 0.6 0.55 0.62 Latency (ms) 19 18 20 In addition, the comparative evaluation of different models in Python is presented as follows: Model Accuracy (%) TF Accuracy (%) PT F1-score TF F1-score PT Latency (ms) TF Latency (ms) PT Evaluation CNN-only 91.2 90.8 0.90 0.89 18 18 Visual-only, limited GRU-only 85.6 85.1 0.84 0.83 15 15 Weak visual capability CNN–Transformer 93.8 93.5 0.93 0.92 25 25 High accuracy, computationally heavy Late Fusion CNN–GRU 94.5 94.2 0.94 0.93 22 22 Moderate performance Hybrid CNN–GRU 95.8 94.6 0.96 0.94 18 20 Most optimal balance The analysis demonstrates that the Hybrid CNN–GRU model successfully integrates spatial and sequential features through cross-attention and feature alignment mechanisms. Training dynamics confirm stable learning, while performance metrics indicate high accuracy and efficiency. Its comparative advantage over unimodal, CNN–Transformer, and late fusion models is evident, particularly in the combined achievement of accuracy, low latency, and robust F1-score. Overall, the experimental evaluation confirms that the Hybrid CNN–GRU model is highly effective, platform-independent, and suitable for real-time multimodal AI applications, including traffic monitoring, autonomous systems, and industrial or smart city surveillance. TensorFlow/Keras offers a faster and more stable practical deployment, while PyTorch provides flexibility for research-driven experimentation. The model achieves approximately 95–96% accuracy in real-time scenarios, validating both its theoretical design and practical implementation. 5. Conclusion The experimental evaluation of the Hybrid CNN–GRU model, implemented in both Python (TensorFlow/Keras and PyTorch) and MATLAB (Deep Learning Toolbox), confirms its effectiveness for real-time multimodal decision-making through the synchronized processing of visual and textual inputs. Across all conducted simulations, the model consistently outperformed single-modal approaches, including CNN-only and GRU-only models, as well as late fusion and CNN–Transformer architectures. In Python, the Hybrid CNN–GRU achieved an accuracy of 95–96% in TensorFlow/Keras and 94–95% in PyTorch, with F1-scores of 0.96 and 0.94, respectively, while maintaining low latency (18–20 ms per prediction), confirming its suitability for real-time applications. MATLAB simulations similarly demonstrated stable convergence and high performance, with validation accuracy ranging from 94% to 96% and a final loss of approximately 0.6. The superior performance of the proposed model is attributed to its hybrid architecture, which effectively extracts spatial features from visual inputs via CNN layers and models sequential dependencies in textual data using GRU layers. The cross-attention and feature alignment mechanisms enable seamless integration of multimodal information, allowing the model to capture nuanced patterns such as traffic rule violations more reliably than unimodal or post-fusion methods. Comparative analysis shows that the Hybrid CNN–GRU model achieves the most optimal balance between predictive accuracy, computational efficiency, and robustness, outperforming alternative architectures in both accuracy and latency. Furthermore, SHAP-based interpretability analysis in Python confirmed that the model transparently exploits interactions between visual and textual modalities, enabling explainable predictions and supporting trustworthiness in safety-critical real-time systems. The flexibility of the model in PyTorch allows for research-oriented experimentation and detailed optimization, whereas TensorFlow/Keras provides faster convergence and practical deployment advantages. In conclusion, the Hybrid CNN–GRU framework, validated across multiple software platforms, demonstrates a robust, scalable, and interpretable solution for real-time multimodal AI applications. Its consistent high accuracy, low latency, and stable learning behavior make it particularly suitable for smart city management, traffic surveillance, industrial safety monitoring, and autonomous robotic systems, offering both theoretical and practical efficiency in diverse real-time operational scenarios. Declarations Consent to Publish: Not applicable. Author Contribution A.M. (Aida Mustafayeva) conceptualized and designed the study, supervised the research, and revised the manuscript.E.I. (Elmira Israfilova) developed the hybrid CNN–GRU model, performed experiments, and analyzed the results.G.B. (Gunel Baxshiyeva) prepared the figures, tables, and data visualization.S.A. (Saadat Aslanova) contributed to data preprocessing, simulation, and manuscript drafting.All authors reviewed and approved the final version of the manuscript. Data Availability Data Availability Statement (Optimal Version):Yes. The datasets used and/or analyzed during the current study are publicly available. The traffic image dataset can be accessed at https://www.kaggle.com/datasets/meowmeowmeowmeowmeow/gtsrb-german-traffic-sign. Any additional data supporting the findings of this study are available from the corresponding author upon reasonable request. References Antol S, Agrawal A, Lu J, Mitchell M, Batra D, Zitnick L, C., Parikh D. (2015). VQA: Visual Question Answering. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2425–2433. https://doi.org/10.1109/ICCV.2015.279 Acosta JN, Falcone GJ, Rajpurkar P, Topol EJ. Multimodal biomedical artificial intelligence. Nat Med. 2022;28:1773–84. https://doi.org/10.1038/s41591-022-01981-2 . Binte Rashid M, Rahaman MS, Rivas P. (2024). Navigating the Multimodal Landscape: A Review on Integration of Text and Image Data. Machine Learning and Knowledge Extraction. https://doi.org/10.3390/make6030074 Chen X, Xie H, Tao X, Wang FL, Leng M, Lei B. (2024). Artificial intelligence and multimodal data fusion for smart healthcare: topic modeling and bibliometrics. Artificial Intelligence Review (Springer). https://doi.org/10.1007/s10462-024-10712-7 Dixit C, Satapathy SM. Deep CNN with late fusion for real-time multimodal emotion recognition. Expert Syst Appl. 2024;240., Article 122579. https://doi.org/10.1016/j.eswa.2023.122579 . Hao X, Du H, Guo J et al. (2025). A CNN–Transformer Hybrid Model for Multimodal Person Re-Identification. International Journal of Multimedia Information Retrieval. https://doi.org/10.1007/s13735-025-00367-7 Huang M, Jia S, Chang M-C, Lyu S. Text-image de-contextualization detection using vision-language models. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2022) , Virtual, 7–13 May 2022. Gupta C, Gill NS, Gulia P et al. (2025). A multimodal fusion model for real-time emotion recognition using audio-visual-textual features. Journal of Big Data (Springer). https://doi.org/10.1186/s40537-025-01300-9 Liu Y, Zhu X, Clifton DA. Multimodal Learning with Transformers: A Survey. IEEE Trans Pattern Anal Mach. 2023. https://doi.org/10.1109/TPAMI.2023.3275156 . Intelligence (TPAMI). Li G, Ren G, Wang J, Yu Z, Jiang B, Guo Q. (2025). Multimodal fusion transformer network for multispectral pedestrian detection in low-light condition. Scientific Reports (Nature). https://doi.org/10.1038/s41598-025-03567-7 Liu Y. (2024). Multimodal NLP and Cross-Media Information Understanding. Proceedings of SDMC 2024. https://doi.org/10.2991/978-2-38476-327-6_24 Makhmudov F, Kultimuratov A, Cho Y. Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures. Appl Sci. 2024;14(10):4199. https://doi.org/10.3390/app14104199 . Meel P, Vishwakarma DK. Multi-modal fusion using fine-tuned self-attention and transfer learning for veracity analysis of web information. Expert Syst Appl. 2023;229:120537. Nakach F-Z, Idri A, Goceri E. A comprehensive investigation of multimodal deep learning fusion strategies for breast cancer classification. Artif Intell Rev Springer DOI. 2024. https://doi.org/10.1007/s10462-024-10984-z . Rasheed J, Jamil A, Hasibe B. Turkish Text Detection System from Videos Using Machine Learning and Deep Learning Techniques. IEEE Third International Conference on Data Stream Mining & Processing August 21–25, 2020, Lviv, Ukraine. 10.1109/DSMP47368.2020.9204036 Shi D, Zhang W, Yang J et al. (2025).A multimodal vision–language foundation model for computational medicine. npj Digital Medicine. https://doi.org/10.1038/s41746-025-01772-2 Shaikh MB, Islam SMS, Chai D, Akhtar N. Multimodal fusion for audio-image and video action recognition. Neural Comput Appl. 2024;Q1. https://doi.org/10.1007/s00521-023-09186-5 . Shao W, Fan D, Cui C et al. (2026). Deep learning-based astronomical multimodal data fusion. https://doi.org/10.1016/j.inffus.2025.104103 Tsai YHH, Bai S, Yamada M, Morency LP, Salakhutdinov R. (2019). Multimodal Transformer for Multimodal Sentiment Analysis. Proceedings of the ACL. https://doi.org/10.18653/v1/P19-1623 Wang J-H, Norouzi M, Tsai SM. Augmenting Multimodal Content Representation with Transformers for Misinformation Detection. Big Data Cogn Comput. 2024;8(10):134. https://doi.org/10.3390/bdcc8100134 . Wang H. (2024). Multimodal Audio-Visual Fusion Using 3D CNN and CRNN for Behavior Recognition. Frontiers in Neurorobotics. https://doi.org/10.3389/fnbot.2024.1284175 Xu P, Zhu X, Clifton DA. Multimodal Learning with Transformers: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). https://doi.org/10.1109/TPAMI.2023.3275156 Vinyals O, Toshev A, Bengio S, Erhan D. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , Boston, MA, USA, 7–12 June 2015; pp. 3156–3164. Zhao Y, Mamat M, Aysa A et al. (2023). Multimodal Sentiment System Based on CRNN-SVM. Neural Computing and Applications. https://doi.org/10.1007/s00521-023-08366-7 Zengyi Yang Y, Li X, Tang et al. (2024). MGFusion: A multimodal large language model-guided framework for image fusion. Frontiers in Neurorobotics. https://doi.org/10.3389/fnbot.2024.1521603 Zhang D, Wong WK, Chew IM. (2025). A comprehensive review of multimodal visual representation learning: tracing the evolution from CNNs to transformers and beyond. International Journal of Multimedia Information Retrieval (Springer). https://doi.org/10.1007/s13735-025-00382-8 Yang Z, Li Y, Tang X, Xie M. MGFusion: A multimodal large language model-guided framework for image fusion. Front Neurorobotics. 2024;18. https://doi.org/10.3389/fnbot.2024.1521603 . Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 13 May, 2026 Reviews received at journal 25 Apr, 2026 Reviews received at journal 23 Apr, 2026 Reviewers agreed at journal 20 Apr, 2026 Reviewers agreed at journal 19 Apr, 2026 Reviewers agreed at journal 17 Apr, 2026 Reviews received at journal 17 Apr, 2026 Reviewers agreed at journal 17 Apr, 2026 Reviewers agreed at journal 17 Apr, 2026 Reviewers agreed at journal 17 Apr, 2026 Reviewers invited by journal 17 Apr, 2026 Editor assigned by journal 31 Mar, 2026 Submission checks completed at journal 31 Mar, 2026 First submitted to journal 29 Mar, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9257523","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":628420611,"identity":"069d6e12-6ae6-4802-bd96-b60506e991e8","order_by":0,"name":"Aida Mustafayeva","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABKklEQVRIiWNgGAWjYDACZgYDKIvxAZjiZwCLMBPWwsPADGFINhDSwoCuxeAAAS387cwbPzDmHLa3Zz/M+Jmn4rC88bXD2yQYKqwTG9iPP8CmReIwW7EE47bDiT08yczSPGcOG267nVYmwXAmPbGBJyEBqzWHeQxAWhJ4GPIPSPO2pTFuu51jJsHYdjixgSHhADYd8od5jH8Atdjz8D9m/s37L81+82yQln9ALfwPG7D6/TCPGcgWxh6JZDZp3gabxA3SIC0NQC0SyVjdZXiYrcwicVt6Ys+Nx2yWc47ZJM+4nVZskXAs3bhN4hlWLXLnD2++8XGbtT17fzLzjTc1Erb9s5M33vhQYy3bz5+ONcTAABYwTDzIImw41SMBxh/EqBoFo2AUjIIRBwAzCV2evgS77QAAAABJRU5ErkJggg==","orcid":"","institution":"Mingachevir State University","correspondingAuthor":true,"prefix":"","firstName":"Aida","middleName":"","lastName":"Mustafayeva","suffix":""},{"id":628420612,"identity":"b020110d-b065-4ac5-98a8-37c1c2aab180","order_by":1,"name":"Elmira Israfilova","email":"","orcid":"","institution":"Mingachevir State University","correspondingAuthor":false,"prefix":"","firstName":"Elmira","middleName":"","lastName":"Israfilova","suffix":""},{"id":628420614,"identity":"0a88cf92-b37b-4b0d-864f-91414f402cbb","order_by":2,"name":"Gunel Baxshiyeva","email":"","orcid":"","institution":"Mingachevir State University","correspondingAuthor":false,"prefix":"","firstName":"Gunel","middleName":"","lastName":"Baxshiyeva","suffix":""},{"id":628420616,"identity":"656c6099-c044-42ff-a747-cdb659221dbd","order_by":3,"name":"Saadat Aslanova","email":"","orcid":"","institution":"Mingachevir State University","correspondingAuthor":false,"prefix":"","firstName":"Saadat","middleName":"","lastName":"Aslanova","suffix":""}],"badges":[],"createdAt":"2026-03-29 08:53:21","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9257523/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9257523/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":107708048,"identity":"3bb899cf-bb62-46f2-884c-21df909da420","added_by":"auto","created_at":"2026-04-24 09:21:45","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":201838,"visible":true,"origin":"","legend":"\u003cp\u003eFig. 3.1. Functional architecture of the proposed hybrid CNN-GRU multimodal processing\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-9257523/v1/15dcebdd52bf1ea51a4973f1.png"},{"id":107707961,"identity":"bbbe9ea7-8b83-4e4e-b8ed-1015a8acb703","added_by":"auto","created_at":"2026-04-24 09:21:31","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":658622,"visible":true,"origin":"","legend":"\u003cp\u003eFig.3.2. Processing of Visual Data via CNN\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-9257523/v1/9df3e565430fd0ed1cd1df9e.png"},{"id":107707105,"identity":"6ff8222f-cb4e-4c9c-9187-57e062951f5d","added_by":"auto","created_at":"2026-04-24 09:19:31","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":600942,"visible":true,"origin":"","legend":"\u003cp\u003eFig. 3.3. \u003cstrong\u003eProcessing Textual Data via GRU\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-9257523/v1/ac04363b257edc5f3ad890da.png"},{"id":107706872,"identity":"fb2cee6d-cda4-4c0b-bf7e-15dd2414e318","added_by":"auto","created_at":"2026-04-24 09:18:58","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":749106,"visible":true,"origin":"","legend":"\u003cp\u003eFig. 3.4. Multimodal Feature Integration\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-9257523/v1/debd3bccd91ff3f0aa6a9a10.png"},{"id":107696943,"identity":"0eec6972-dc0d-48ce-bb49-8c060dc1eaae","added_by":"auto","created_at":"2026-04-24 07:18:39","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":247659,"visible":true,"origin":"","legend":"\u003cp\u003eFig. 4.1. \u003cstrong\u003eGTSRB \u003c/strong\u003eDataset structure\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-9257523/v1/ca506e275674dd5b7da95d6a.png"},{"id":107696946,"identity":"91f00cfe-4e54-4157-92c6-b31912e87ef2","added_by":"auto","created_at":"2026-04-24 07:18:39","extension":"jpeg","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":741860,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eFig.4.2. Model Performance Metrics Graphics\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"6.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-9257523/v1/ce81c52495ea334b589dab13.jpeg"},{"id":107696948,"identity":"143d9ab7-e754-4b4a-a5a8-6e3076d8d3bb","added_by":"auto","created_at":"2026-04-24 07:18:39","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":121648,"visible":true,"origin":"","legend":"\u003cp\u003eFig. 4.3. Training and validation Loss\u003c/p\u003e","description":"","filename":"7.png","url":"https://assets-eu.researchsquare.com/files/rs-9257523/v1/afcee0ee27c881521d2fcf27.png"},{"id":107709346,"identity":"ec1593b6-166c-4201-87a6-4dd65c800869","added_by":"auto","created_at":"2026-04-24 09:35:30","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":4037636,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9257523/v1/af32f475-1978-4bf0-a993-d5e5d6db6699.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"\u003cp\u003eHybrid Cnn-gru Model for Real-time Multimodal Decision-making in Image and Text Analysis\u003c/p\u003e","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eIn recent years, multimodal artificial intelligence (AI) systems have witnessed rapid advancements across information technology, healthcare, robotics, and intelligent transportation domains (Acosta et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Chen et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Dixit \u0026amp; Satapathy, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). While traditional approaches primarily processed single-modal data, such as text or images, modern applications increasingly demand the synchronous integration of visual, textual, and sensor modalities (Hao et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Wang, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). This paradigm enables more accurate disease diagnosis in medical imaging, reliable misinformation detection on social media, effective human\u0026ndash;robot interaction, and behavioral analytics (Antol et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Li et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Shao et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2026\u003c/span\u003e). For instance, joint analysis of medical images and clinical records facilitates early disease detection, whereas multimodal video-audio analysis in sports contexts allows precise prediction of player behavior (Wang, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eA primary challenge in multimodal AI lies in the effective integration of heterogeneous data. Each modality differs in scale, structure, and frequency, which can lead to information loss and increased model complexity during fusion (Meel \u0026amp; Vishwakarma, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Liu et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Additionally, real-time analytics in large-scale datasets impose significant computational and performance constraints on existing models (Gupta et al., \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Shaikh et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). In online environments and social media, data decontextualization, manipulation, and rumor propagation present further obstacles for multimodal systems (Tsai et al., \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Huang et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2022\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eDeep learning architectures including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs/GRUs), and transformer-based models such as BERT, ALBERT, and multimodal transformers have been widely employed for multimodal data processing (Devlin et al., 2019; Liu et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Vaswani et al., 2017). Transformer models, leveraging self-attention mechanisms, learn contextual dependencies across modalities, thereby enhancing multimodal integration (Huang et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Meel \u0026amp; Vishwakarma, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Tsai et al., \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). Studies have demonstrated the effectiveness of different multimodal fusion strategies including early fusion, late fusion, and hybrid approaches in real-world applications. These methods have shown superior performance in emotion recognition, medical prognostics, rumor detection on social media, and behavioral analysis (Hao et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Shaikh et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Wang, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Gupta et al., \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2025\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eNevertheless, most transformer-based multimodal models require substantial computational resources and memory, limiting their real-time applicability, particularly in edge devices, traffic monitoring, and industrial safety systems (Li et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Shaikh et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Such scenarios demand multimodal architectures that provide low latency, energy efficiency, and reliable performance.\u003c/p\u003e \u003cp\u003eTraffic violation detection exemplifies these challenges. Visual systems encounter occlusions, varying illumination, and complex dynamic environments, whereas textual information from incident reports and logs provides semantic context for observed behaviors (Antol et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Hao et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). However, many existing systems process visual and textual inputs separately, failing to capture the complementary relationships between motion patterns and semantic descriptions fully.\u003c/p\u003e \u003cp\u003eThis study proposes a lightweight hybrid CNN\u0026ndash;GRU multimodal neural architecture for synchronous processing of visual and textual data streams. The model utilizes CNNs for visual feature extraction and GRUs for sequential textual embedding, achieving a balance between computational efficiency and predictive accuracy compared to transformer-based multimodal models.\u003c/p\u003e \u003cp\u003eExperimental evaluations on multimodal datasets demonstrate that the CNN\u0026ndash;GRU architecture maintains high prediction accuracy while reducing latency. Compared with single-modal approaches, the proposed model improves detection performance, and compared with transformer-based architectures, it achieves lower computational overhead. Furthermore, the integration of SHAP-based explainability mechanisms allows the contributions of visual and textual modalities to be transparently interpreted by human operators (Antol et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Meel \u0026amp; Vishwakarma, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). This approach aligns with the Industry 5.0 paradigm, emphasizing human-centered and trustworthy AI applications.\u003c/p\u003e \u003cp\u003e \u003cb\u003eMain Contributions\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe main scientific contributions of this research can be summarized as follows.\u003c/p\u003e \u003cp\u003eFirst, a lightweight hybrid multimodal architecture combining convolutional neural networks and gated recurrent units is proposed for synchronous processing of visual and textual data. Unlike transformer-based multimodal frameworks, the proposed design focuses on computational efficiency and low-latency operation, making it suitable for real-time monitoring systems deployed in resource-constrained environments.\u003c/p\u003e \u003cp\u003eSecond, the study introduces an efficient multimodal feature integration mechanism that combines spatial visual representations and semantic textual embeddings into a unified decision space. This integration enables the model to capture complementary contextual information across modalities and reduces ambiguity in visually complex monitoring scenarios.\u003c/p\u003e \u003cp\u003eThird, the proposed framework incorporates an explainable multimodal decision pipeline using SHAP-based attribution analysis, allowing the contribution of visual and textual modalities to be interpreted and verified by human operators. This transparency supports trustworthy AI deployment in safety-critical environments.\u003c/p\u003e \u003cp\u003eFourth, an experimental benchmarking study is conducted for multimodal traffic rule violation detection, demonstrating that the proposed CNN\u0026ndash;GRU architecture achieves a favorable balance between predictive performance and computational efficiency compared with CNN-only, GRU-only, and CNN\u0026ndash;Transformer baselines.\u003c/p\u003e \u003cp\u003eFinally, the proposed model provides a scalable and adaptable framework for intelligent multimodal monitoring applications beyond traffic analysis, including industrial safety supervision, smart city surveillance, and automated decision-support systems aligned with Industry 5.0 principles.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePaper Organization\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe remainder of this paper is organized as follows. Section \u003cspan refid=\"Sec2\" class=\"InternalRef\"\u003e2\u003c/span\u003e presents a review and analytical comparison of contemporary approaches to multimodal processing of visual and textual data, highlighting existing methodological limitations and research gaps. Section \u003cspan refid=\"Sec7\" class=\"InternalRef\"\u003e3\u003c/span\u003e describes the functional scheme and software implementation of the proposed hybrid multimodal neural network model. Section \u003cspan refid=\"Sec10\" class=\"InternalRef\"\u003e4\u003c/span\u003e introduces the proposed methodology and the architecture of the CNN\u0026ndash;GRU framework, including the mathematical formulation and multimodal fusion mechanism. Section \u003cspan refid=\"Sec16\" class=\"InternalRef\"\u003e5\u003c/span\u003e provides the experimental evaluation, including dataset description, training configuration, baseline comparisons, and quantitative performance analysis. Finally, Section 6 concludes the paper by summarizing the main findings, discussing practical implications, and outlining potential directions for future research.\u003c/p\u003e"},{"header":"2. Review and Analysis of Contemporary Approaches to the Synchronous Processing of Visual and Textual Data","content":"\u003cp\u003eThe synchronous processing of visual and textual data has emerged as a central challenge in multimodal artificial intelligence, driven by applications in healthcare, intelligent transportation, human\u0026ndash;robot interaction, and social media analysis (Acosta et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Chen et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Dixit \u0026amp; Satapathy, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Integrating these heterogeneous data streams enhances contextual understanding, improves predictive performance, and reduces ambiguities inherent in single-modal systems (Antol et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Li et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Shao et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2026\u003c/span\u003e)\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Convolutional and Recurrent Neural Network Approaches\u003c/h2\u003e \u003cp\u003eEarly multimodal integration methods relied on Convolutional Neural Networks (CNNs) for visual feature extraction and Recurrent Neural Networks (RNNs)/Gated Recurrent Units (GRUs) for sequential textual processing \u003cb\u003e(\u003c/b\u003eTable\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e2.1\u003c/span\u003e). These architectures, combined through late or hybrid fusion strategies, effectively model the spatial-temporal dependencies of visual-textual streams while maintaining moderate computational efficiency (Dixit \u0026amp; Satapathy, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Wang, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Hao et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2025\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eFor instance, Wang (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) demonstrated that 3D CNNs combined with CRNNs improved behavioral recognition in video-audio datasets, providing richer temporal-spatial representations than unimodal alternatives. Similarly, attention-enhanced CNN\u0026ndash;BERT architectures significantly increased emotion recognition accuracy in multimodal datasets (Makhmudov et al., \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). However, these approaches face challenges in modeling long-range contextual interactions across modalities. Late fusion methods may overlook complementary information during integration, resulting in suboptimal joint representations (Gupta et al., \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Shaikh et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2.1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eGrouped Analysis by Model Architecture \u0026ndash; CNN \u0026amp; RNN/GRU Approaches\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel Type\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStudies\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFusion Strategy\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eKey Domains\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eObservations\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;RNN/GRU\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1, 5, 8, 17, 21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLate/Hybrid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eEmotion Recognition, Action Recognition, Behavior Analysis\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eEffective for sequential modeling; moderate computational efficiency; captures local/spatial-temporal features.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;BERT / Attention\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHybrid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eEmotion Recognition\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eAttention enhances integration; improved interpretability; moderate computational cost.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;CRNN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHybrid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eBehavior Recognition\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eTemporal-spatial features captured efficiently; outperforms unimodal baselines.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Transformer-Based Multimodal Architectures\u003c/h2\u003e \u003cp\u003eTransformers, with self-attention mechanisms, have revolutionized synchronous multimodal processing by learning global dependencies and cross-modal interactions (Vaswani et al., 2017; Liu et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Tsai et al., \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2019\u003c/span\u003e) \u003cb\u003e(\u003c/b\u003eTable\u0026nbsp;2b).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2.2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eGrouped Analysis by Model Architecture \u0026ndash; Transformer Approaches\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel Type\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStudies\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFusion Strategy\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eKey Domains\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eObservations\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTransformer-based\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e3, 4, 9, 10, 11, 16, 20, 22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eEarly/Hybrid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHealthcare, NLP, Pedestrian Detection, Computational Medicine\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCaptures global dependencies; attention enables semantic alignment; resource-intensive.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;Transformer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e6, 14, 18, 26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHybrid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePerson Re-ID, Breast Cancer Classification, Astronomy, Visual Representation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCombines local visual features with global reasoning; highest performance across complex datasets; high computational cost.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLarge Multimodal LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e25, 27\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHybrid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eImage Fusion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eScalable cross-domain multimodal fusion; emerging trend for generalized reasoning.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eModels such as \u003cb\u003eCLIP, ViLT, and multimodal BERT variants\u003c/b\u003e align textual embeddings with visual feature spaces, enabling semantic grounding of images through textual descriptions (Antol et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Liu, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Binte Rashid et al., \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Recent studies highlight superior performance of transformer-based architectures. Shaikh et al. (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) proposed a multimodal fusion model integrating audio, visual, and textual inputs for action recognition, showing that attention-based alignment improves prediction accuracy. Hao et al. (\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) developed a \u003cb\u003eCNN\u0026ndash;Transformer hybrid\u003c/b\u003e for person re-identification, combining local visual patterns from CNNs with global reasoning from transformers Nevertheless, these architectures are resource-intensive and may not meet real-time requirements in edge devices or low-latency environments (Li et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Zhang et al., \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Huang et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2022\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Multimodal Fusion Strategies\u003c/h2\u003e \u003cp\u003eMultimodal data integration approaches can be categorized into early fusion, late fusion, and hybrid strategies (Xu et al., 2023; Nakach et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Zhao et al., \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Early fusion combines raw features from different modalities into a unified representation, which is particularly effective for tightly aligned visual-textual data but is highly sensitive to noise and increases model complexity. Late fusion, in contrast, aggregates predictions from independently trained unimodal networks, preserving modality-specific patterns and simplifying optimization, yet it may overlook complementary cross-modal information that could enhance joint representations. Hybrid fusion integrates intermediate representations using attention-based weighting mechanisms, striking a balance between robustness, cross-modal interaction, and predictive accuracy, making it especially suitable for heterogeneous datasets and complex real-world tasks (Li et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Makhmudov et al., \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Shaikh et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e2.3\u003c/span\u003e.).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2.3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eGrouped Analysis by Fusion Strategy\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFusion Type\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStudies\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eModel Types\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eDomains\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eObservations\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEarly Fusion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1, 3, 11, 15, 23\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;LSTM, Transformer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eVQA, Image Captioning, NLP, Text Detection\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCaptures raw cross-modal interactions; limited robustness to noise; moderate complexity.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLate Fusion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e5, 13, 17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;RNN, Transformer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eEmotion Recognition, Action Recognition\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSimplifies optimization; preserves unimodal patterns; may miss inter-modal info.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHybrid Fusion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 21, 24, 25, 26, 27\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;Transformer, Transformer, CNN\u0026thinsp;+\u0026thinsp;BERT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHealthcare, Re-ID, Astronomy, Image Fusion, Behavior Analysis\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eBalances early \u0026amp; late fusion benefits; highest performance; computationally demanding; supports explainability.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e2.4 Application Domains and Observed Trends\u003c/h2\u003e \u003cp\u003eIn healthcare and biomedical applications, CNN\u0026ndash;Transformer and transformer-only models employing hybrid fusion significantly improve early diagnosis accuracy, enhance interpretability, and utilize attention mechanisms to support semantic reasoning (Acosta et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Chen et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Shi et al., \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). Emotion recognition and behavioral analysis rely on CNN\u0026thinsp;+\u0026thinsp;RNN/GRU, CNN\u0026thinsp;+\u0026thinsp;BERT, and CRNN-SVM architectures, where late and hybrid fusion capture critical temporal-spatial dependencies across audio, visual, and textual streams, resulting in high recognition performance (Dixit \u0026amp; Satapathy, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Makhmudov et al., \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Gupta et al., \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). In smart cities, pedestrian monitoring, and traffic analysis, transformer-based models with hybrid or early fusion strategies support real-time detection under challenging conditions such as low-light or occlusions, although latency and computational efficiency remain practical constraints (Li et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Wang et al., \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Image captioning, visual question answering, and NLP tasks typically use CNN\u0026thinsp;+\u0026thinsp;LSTM or transformer models with early fusion, which works well for tightly coupled visual-textual data but exhibits moderate sensitivity to noisy datasets (Antol et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Liu, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Person re-identification and multimedia retrieval benefit from CNN\u0026ndash;Transformer hybrid models that align cross-modal features to achieve high retrieval accuracy (Hao et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Zhang et al., \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). Emerging domains such as astronomy and multimodal image fusion employ CNN\u0026thinsp;+\u0026thinsp;Transformer models or large multimodal LLMs with hybrid fusion, enabling generalized reasoning across heterogeneous data sources and opening avenues for new applications (Shao et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2026\u003c/span\u003e; Yang et al., \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Zengyi et al., 2024). In social media and misinformation detection, transformers and vision-language models leverage hybrid fusion to enhance semantic alignment and interpretability, which is crucial for user trust and practical deployment (Huang et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Tsai et al., \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Wang, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Overall, these application-specific trends reveal the critical role of hybrid fusion in maximizing model performance, while highlighting ongoing challenges related to computational efficiency, latency, and domain generalization (Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e2.4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2.4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eGrouped Analysis by Application Domain\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDomain\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStudies\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCommon Models\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eFusion Strategies\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eObservations\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHealthcare / Biomedical\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2, 4, 14, 16\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;Transformer, Transformer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHybrid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eImproves early diagnosis accuracy; interpretable transformers enhance trust; attention boosts semantic reasoning.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEmotion Recognition / Behavior\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e5, 8, 12, 17, 21, 24\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;RNN/GRU, CNN\u0026thinsp;+\u0026thinsp;BERT, CRNN-SVM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eLate / Hybrid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eTemporal-spatial modeling critical; hybrid fusion captures audio-visual-textual cues; high accuracy achieved.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSmart Cities / Pedestrian / Traffic\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e10, 20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTransformer, Multimodal Fusion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHybrid / Early\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eReal-time monitoring; hybrid fusion improves detection under low-light and occlusions; latency remains a challenge.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eImage Captioning / VQA / NLP\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1, 11, 23\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;LSTM, Transformer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eEarly Fusion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eEarly fusion suitable for tightly coupled visual-text data; moderate performance under noisy datasets.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePerson Re-ID / Multimedia Retrieval\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e6, 26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;Transformer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHybrid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSupports cross-modal feature alignment; high retrieval accuracy.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAstronomy / Image Fusion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e18, 25, 27\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;Transformer, Large Multimodal LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHybrid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCross-source data fusion; LLMs enable generalized multimodal reasoning; emerging applications.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eLiterature analysis indicates that in contemporary multimodal artificial intelligence applications, transformer-based and hybrid CNN\u0026ndash;Transformer models demonstrate superior performance in healthcare, visual data fusion, and person re-identification tasks (Hao et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Li et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Shaikh et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Conversely, CNN combined with RNN/GRU architectures remain effective for sequential tasks, particularly in emotion and behavior recognition, highlighting the continued relevance of classical deep learning approaches for capturing local and temporal dependencies in visual-textual data (Dixit \u0026amp; Satapathy, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Wang, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Makhmudov et al., \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eRegarding fusion strategies, studies show that hybrid fusion provides the most balanced and reliable performance across heterogeneous multimodal datasets by leveraging the advantages of both early and late integration methods (Li et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Shaikh et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Makhmudov et al., \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Early fusion is particularly effective for tightly coupled visual-text pairs, as it allows cross-modal interactions at the initial stage. In contrast, late fusion is better suited for sequential tasks such as video-audio analysis and behavior recognition (Wang, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Dixit \u0026amp; Satapathy, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eField observations indicate that in healthcare and emotion recognition applications, hybrid fusion not only enhances model explainability but also facilitates learning contextual cause-and-effect relationships between visual and textual modalities (Acosta et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Antol et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2015\u003c/span\u003e). Furthermore, emerging domains such as astronomy, smart cities, and large multimodal LLM applications suggest promising research directions for hybrid transformer models and multimodal large language models (LLMs) (Shao et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2026\u003c/span\u003e; Binte Rashid et al., \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Yang et al., \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Nevertheless, several challenges remain unresolved. Real-time applications are constrained by computational costs and latency, limiting deployment on edge devices (Li et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Shaikh et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Cross-modal alignment in noisy and incomplete data remains an open problem, and explainability and cross-domain generalization continue to require focused investigation (Antol et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Binte Rashid et al., \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). These factors highlight the necessity for future research to develop hybrid multimodal architectures that optimize computational efficiency, model interpretability, and domain transferability.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. Functional Architecture and Implementation of a Hybrid CNN–GRU Framework for Multimodal Data Processing","content":"\u003cp\u003eThe processing of multimodal data is considered one of the most important research directions in modern artificial intelligence systems. This approach involves the simultaneous analysis of diverse data sources\u0026mdash;such as visual information (images, video frames) and textual information (reports, descriptions, sensor logs)\u0026mdash;and their integration within a unified decision-making mechanism. While traditional unimodal systems operate on a single type of data, multimodal approaches leverage the complementary characteristics of information from different modalities, enabling more accurate and context-aware outcomes. This capability is particularly critical in applications such as real-time monitoring systems, smart city platforms, human\u0026ndash;robot interaction, and behavioral analytics. The multimodal data processing pipeline generally consists of several key stages, which define the functional structure of the system architecture (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3.1\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003eThe presented diagram illustrates the functional architecture of a hybrid CNN\u0026ndash;GRU multimodal neural network model designed for the synchronous processing of visual and textual data. This model enables the parallel analysis of heterogeneous data sources and their integration within a unified decision-making mechanism. The multimodal processing pipeline comprises several sequential stages:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\n\u003cp\u003eInput Layer\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eFeature Extraction\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eMultimodal Feature Integration (Hybrid Fusion Module)\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eMultimodal Decision Layer\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003ePrediction Output\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eExplainable Artificial Intelligence\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eEfficient Real-Time Deployment\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eAt the Input Layer, the process begins with the acquisition of data from multiple modalities. Two primary data types are considered: Visual Data \u0026ndash; Images and video frames obtained from surveillance cameras and sensor systems, providing spatial information regarding object locations, movements, and environmental visual characteristics. Textual Data \u0026ndash; Semantic information such as incident reports, system logs, and descriptive log files, which provide contextual explanations of observed events. At this stage, the data is ingested and prepared for subsequent processing.\u003c/p\u003e\n\u003cp\u003eIn the Feature Extraction stage, each modality is processed using an appropriate deep learning model: Visual Feature Extraction \u0026ndash; CNN Convolutional Neural Networks (CNNs) process visual data through multi-level convolution and pooling operations, detecting object contours, motion patterns, and other spatial attributes. The resulting output is a high-level visual feature vector representing the image data. Textual Feature Extraction \u0026ndash; GRU\u003c/p\u003e\n\u003cp\u003eGiven the sequential nature of text, a Gated Recurrent Unit (GRU) model is employed. GRU captures semantic relationships and contextual dependencies within textual sequences, converting textual information into a compact semantic embedding vector. This enables effective modeling of the meaning conveyed by event descriptions and system logs.\u003c/p\u003e\n\u003cp\u003eThe features extracted by CNN and GRU are subsequently integrated in the Hybrid Fusion Module, which employs two key mechanisms:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\n\u003cp\u003eCross-Attention Mechanism \u0026ndash; Learns interdependencies between visual and textual features and determines the influence of each modality on the other.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eFeature Alignment \u0026ndash; Aligns feature vectors of varying dimensions and structures into a common feature space.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eAt this stage, the model identifies semantic correspondences between visual events and textual descriptions, forming a unified multimodal representation. The integrated features from the fusion stage are then forwarded to the Multimodal Decision Layer, which consists of fully connected layers and classification mechanisms. Here, information from multiple modalities is jointly analyzed to reach a final decision regarding the observed event or behavior.\u003c/p\u003e\n\u003cp\u003eThe primary output of the system is generated in the Prediction Output module. This module classifies the type of event based on the analysis. For instance, in a traffic monitoring system, the model may output one of the following:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eRule violation detected\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eNormal behavior observed\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eIndeterminate situation\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eTo enhance transparency, the architecture includes a SHAP-based explainability mechanism, which quantifies the contribution of each modality to the decision and facilitates user understanding of the system\u0026rsquo;s outputs.\u003c/p\u003e\n\u003cp\u003eThe proposed CNN\u0026ndash;GRU architecture is computationally efficient, ensuring low latency and high processing speed. These characteristics enable deployment in real-time applications such as:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eSmart city monitoring systems\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eTraffic safety platforms\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eIndustrial safety surveillance\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eBehavioral analytics and robotic monitoring systems\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIn summary, the presented multimodal processing pipeline supports the parallel analysis of visual and textual data, feature extraction via CNN and GRU, integration through a hybrid fusion mechanism, and the generation of explainable decisions. Compared to unimodal models, this approach provides superior accuracy, enhanced contextual understanding, and real-time decision-making capabilities.\u003c/p\u003e\n\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e\n\u003ch2\u003e3.3. Mathematical Formulation of the Multimodal Data Processing Pipeline\u003c/h2\u003e\n\u003cp\u003eThe primary objective of multimodal artificial intelligence systems is to construct a more accurate and reliable decision-making model by integrating features extracted from diverse data modalities, such as visual images and textual information. To achieve this, a hybrid CNN\u0026ndash;GRU architecture is employed: the CNN processes visual data, while the GRU handles textual sequences. Processing of Visual Data via CNN. Consider a color image of a vehicle with dimensions 64\u0026times;64 as input. Color images consist of three channels: Red (R), Green (G), and Blue (B). Each channel of the 64\u0026times;64 image is represented by values in the range 0\u0026ndash;255 (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3.2\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIn Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3.2\u003c/span\u003e, the input for each channel is represented as a 64\u0026times;64\u0026times;3 matrix, defined as:\u003c/p\u003e\n\u003cdiv id=\"Equ1\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equ1\" class=\"mathdisplay\"\u003e$$\\:Iϵ{R}^{64\\times\\:64\\times\\:3}$$\u003c/div\u003e\n\u003cdiv class=\"EquationNumber\"\u003e1\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003ewhere \u003cem\u003eI\u003c/em\u003e denotes the input image tensor, 64\u0026times;64 corresponds to the height and width of the image, and 3 represents the number of color channels (R, G, B). For each channel, there exists a 64\u0026times;64 pixel matrix:\u003c/p\u003e\n\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equa\" class=\"mathdisplay\"\u003e$$\\:\\text{R}-\\text{k}\\text{a}\\text{n}\\text{a}\\text{l}={\\left[\\begin{array}{c}123\\:\\:125\\:\\:130\\dots\\:110\\\\\\:115\\:118\\:121\\dots\\:112\\\\\\:⋮\\:\\:\\:\\:\\:\\:\\:\\:⋮\\:\\:\\:\\:\\:\\:\\:\\:\\:⋮\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\ddots\\:\\\\\\:98\\:\\:101\\:\\:\\:\\:105\\dots\\:\\:99\\end{array}\\right]}_{64\\times\\:64}\\:G-\\text{k}\\text{a}\\text{n}\\text{a}\\text{l}={\\left[\\begin{array}{c}100\\:\\:102\\:\\:107\\dots\\:95\\\\\\:98\\:\\:\\:\\:101\\:104\\dots\\:96\\\\\\:⋮\\:\\:\\:\\:\\:\\:\\:\\:⋮\\:\\:\\:\\:\\:\\:\\:\\:\\:⋮\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\ddots\\:\\\\\\:96\\:\\:93\\:\\:\\:\\:97\\dots\\:\\:89\\end{array}\\right]}_{64\\times\\:64}$$\u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Equb\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equb\" class=\"mathdisplay\"\u003e$$\\:B-\\text{k}\\text{a}\\text{n}\\text{a}\\text{l}={\\left[\\begin{array}{c}80\\:\\:82\\:\\:87\\dots\\:70\\\\\\:78\\:\\:\\:\\:81\\:84\\dots\\:72\\\\\\:⋮\\:\\:\\:\\:\\:\\:\\:\\:⋮\\:\\:\\:\\:\\:\\:\\:\\:\\:⋮\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\ddots\\:\\\\\\:60\\:\\:63\\:\\:\\:\\:67\\dots\\:\\:58\\end{array}\\right]}_{64\\times\\:64}$$\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003eThis tensor is provided as input to the CNN model. The CNN applies a convolution operation to extract local features at each pixel. Let us assume a 3\u0026times;3 filter matrix W(k), where k denotes the index of the filter. The convolution operation is computed as:\u003c/p\u003e\n\u003cdiv id=\"Equ2\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equ2\" class=\"mathdisplay\"\u003e$$\\:{F}_{i,j}^{\\left(k\\right)}=\\sigma\\:\\left(\\sum\\:_{m-0}^{2}\\sum\\:_{n-0}^{2}{W}_{m,n}^{\\left(k\\right)}\\bullet\\:{I}_{i+m,j+n}+{b}_{k}\\right)$$\u003c/div\u003e\n\u003cdiv class=\"EquationNumber\"\u003e2\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{F}_{i,j}^{\\left(k\\right)}\\)\u003c/span\u003e\u003c/span\u003eis the \u003cem\u003e(i,j)\u003c/em\u003e-th element of the feature map obtained from the k-th filter, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{W}_{m,n}^{\\left(k\\right)}\\)\u003c/span\u003e\u003c/span\u003e is the weight of the \u003cem\u003e(m,n)\u003c/em\u003e-th element of the k-th filter, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{I}_{i+m,j+n}\\)\u003c/span\u003e\u003c/span\u003e is the corresponding input image pixel, bk is the bias parameter, and \u0026sigma;(\u0026sdot;) is the activation function (e.g., ReLU). Based on Eq.\u0026nbsp;(\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e), the 3-dimensional filter slides over the \u003cem\u003e64\u0026times;64\u003c/em\u003e input image to extract local features, such as edges and textures, thereby generating the feature map:\u003c/p\u003e\n\u003cdiv id=\"Equ3\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equ3\" class=\"mathdisplay\"\u003e$$\\:\\text{F}=\\text{C}\\text{N}\\text{N}\\left(\\text{I}\\right)\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:$$\u003c/div\u003e\n\u003cdiv class=\"EquationNumber\"\u003e3\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003eIn this process, high activation values appear in the output corresponding to visual patterns detected by the filter, meaning the filter \u0026ldquo;activates\u0026rdquo; more strongly in regions where these features are present. The feature map obtained through convolution and pooling has a 2D/3D structure, for example:\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eF\u0026isin;R\u003c/em\u003e \u003csup\u003e \u003cem\u003e16\u0026times;16\u0026times;32\u003c/em\u003e \u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003ewhere 16\u0026times;16 represents the spatial dimensions (height \u0026times; width) and 32 denotes the number of filters (channels). This map indicates which features are active in different regions of the image. For instance, the contours and wheels of a car exhibit high activation values. The feature map is then transformed from a 2D/3D matrix into a 1D vector via a Flatten operation:\u003c/p\u003e\n\u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e \u003cspan class=\"mathinline\"\u003e\\(\\:\\text{v}=\\text{F}\\text{l}\\text{a}\\text{t}\\text{t}\\text{e}\\text{n}\\left(\\text{F}\\right),\\:\\:\\)\u003c/span\u003e \u003c/span\u003ev\u0026isin;R\u003csup\u003e16\u0026sdot;16\u0026sdot;32=8192\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eHere, each element represents pixel and filter information from the original 2D/3D matrix. The resulting 1D vector serves as input to fully connected layers. The Flatten operation simply arranges all matrix elements sequentially in a single row, enabling the complex features extracted by the CNN to be used for decision-making through fully connected layers.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eProcessing Textual Data via GRU.\u003c/strong\u003e Consider a textual description corresponding to the vehicle image:\u003c/p\u003e\n\u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e \u003cspan class=\"mathinline\"\u003e\\(\\:\\text{T}=(\"\\text{r}\\text{e}\\text{d}\",\"\\text{c}\\text{a}\\text{r}\",\"\\text{o}\\text{n}\",\\)\u003c/span\u003e \u003c/span\u003e\"road\")\u003c/p\u003e\n\u003cp\u003eEach word is first converted into a vector representation:\u003c/p\u003e\n\u003cp\u003e\u003cem\u003ex\u003c/em\u003e \u003csub\u003e \u003cem\u003et\u003c/em\u003e \u003c/sub\u003e \u003cem\u003e=Embedding(w\u003c/em\u003e \u003csub\u003e \u003cem\u003et\u003c/em\u003e \u003c/sub\u003e \u003cem\u003e), x\u003c/em\u003e \u003csub\u003e \u003cem\u003et\u003c/em\u003e \u003c/sub\u003e \u003cem\u003e\u0026isin;R\u003c/em\u003e \u003csup\u003e \u003cem\u003ed\u003c/em\u003e \u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003ewhere w\u003csub\u003et\u003c/sub\u003e denotes the word at position t, and d is the dimensionality of the embedding vector (e.g., 50). For example, the word \u0026ldquo;car\u0026rdquo; is represented as a 50-dimensional real-valued vector that encodes its semantic features (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3.3\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe GRU updates its hidden state at each time step using two gates. The reset gate determines how much of the previous state should be forgotten, while the update gate controls how the new information is combined with the previous state. Thus, the semantic context generated from the sequential words is captured in the final hidden state h\u003csub\u003eT\u003c/sub\u003e\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\n\u003cp\u003eUpdate Gate: \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\text{z}}_{t}={\\sigma\\:}({\\text{W}}_{z}{x}_{t}+{\\text{U}}_{z}{\\text{h}}_{t}-1+{b}_{z})\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eReset Gate: \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\text{r}}_{t}={\\sigma\\:}({\\text{W}}_{r}{x}_{t}+{\\text{U}}_{z}{\\text{h}}_{t}-1+{b}_{r})\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eCandidate Hidden State: \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\stackrel{\\sim}{{\\text{h}}_{t}}=\\text{t}anh\\left({\\text{W}}_{h}{x}_{t}+{\\text{U}}_{h}{(\\text{r}}_{t}\\odot\\:{\\text{h}}_{t}-1\\right)\\)\u003c/span\u003e\u003c/span\u003e)\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eFinal Hidden State: \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\text{h}}_{t}=\\left(1-{\\text{z}}_{t}\\right)\\odot\\:{\\text{h}}_{t}-1+{\\text{z}}_{t}\\odot\\:\\stackrel{\\sim}{{\\text{h}}_{t}}\\:\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003c/div\u003e\n\u003cp\u003eHere, ⊙ element-wise multiplication, \u0026sigma; is the sigmoid activation function, and tanh represents the hyperbolic tangent. The resulting textual features are represented as \u003cem\u003eh\u0026thinsp;=\u0026thinsp;h\u003c/em\u003e\u003csub\u003e\u003cem\u003eT\u003c/em\u003e\u003c/sub\u003e .This vector encodes all the textual information related to the image description.\u003c/p\u003e\n\u003cp\u003eMultimodal Feature Integration.The textual feature vector h (from the GRU) is sequentially concatenated with the visual feature vector \u003cem\u003ev\u003c/em\u003e (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3.4\u003c/span\u003e):\u003c/p\u003e\n\u003cp\u003ef=[v;h]\u003c/p\u003e\n\u003cp\u003ewhere \u003cem\u003e[v;h]\u003c/em\u003e denotes the concatenation operation. The resulting vector \u003cem\u003ef\u003c/em\u003e represents the multimodal representation, encompassing both visual and textual information, and serves as a unified input for the decision-making stage.\u003c/p\u003e\n\u003cp\u003eThe concatenated vector is transformed into the output through a fully connected layer:\u003c/p\u003e\n\u003cdiv id=\"Equc\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equc\" class=\"mathdisplay\"\u003e$$\\:\\text{y}=\\text{S}\\text{o}\\text{f}\\text{t}\\text{m}\\text{a}\\text{x}\\left({\\text{W}}_{f}+{\\text{b}}_{f}\\right);$$\u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Equd\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equd\" class=\"mathdisplay\"\u003e$$\\:\\text{y}=\\text{S}\\text{o}\\text{f}\\text{t}\\text{m}\\text{a}\\text{x}\\left({\\text{W}}_{f}\\left[flatten(CNN\\left(I\\right);GRU(T)\\right]+{b}_{f}\\right);$$\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\text{W}}_{f}\\:\\)\u003c/span\u003e\u003c/span\u003eis the weight matrix, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\text{b}}_{f}\\)\u003c/span\u003e\u003c/span\u003e is the bias, and y is the probability vector, representing, for example, the color, type, or category of the vehicle. The multimodal vector f is converted into probability values for each category, and the category with the highest probability is taken as the model\u0026rsquo;s prediction.\u003c/p\u003e"},{"header":"4. Experimental Evaluation of the Hybrid CNN–GRU Model","content":"\u003cp\u003eThe experimental evaluation of the proposed hybrid CNN\u0026ndash;GRU framework aims to demonstrate the effectiveness of real-time multimodal decision-making through the synchronous processing of visual and textual data. Experiments were conducted using datasets related to traffic rule violations as well as benchmark multimodal emotion recognition datasets (Antol et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Hao et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). Additionally, the datasets employed were obtained from publicly available sources such as Kaggle, the UCI Machine Learning Repository, and other open-access repositories. The analysis was performed comparatively against classical unimodal CNN and GRU models, CNN\u0026ndash;Transformer hybrid architectures, and late fusion approaches, highlighting the advantages of the proposed hybrid model.\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Experimental Setup and Simulation Environment\u003c/h2\u003e \u003cp\u003eThe hybrid CNN-GRU model was implemented in Python using the TensorFlow and Keras libraries. In MATLAB, the Deep Learning Toolbox was utilized to configure the CNN feature extraction and GRU sequence embedding modules. Experiments were conducted in both environments, enabling real-time simulation of the training, validation, and testing phases. For each data sample, output predictions and latency measurements were evaluated. The datasets were partitioned into 80% for training, 10% for validation, and 10% for testing. The key hyperparameters are presented in Table\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e4.1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eFor the visual inputs, data augmentation was applied (including random rotations, resizing, and brightness adjustments) to enhance the model\u0026rsquo;s generalization capability. Textual inputs were tokenized and sequentially fed into the GRU layers using pre-trained 300-dimensional GloVe embeddings. Experiments were evaluated both in terms of real-time simulation and CPU/GPU performance.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4.1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eExperimental Configuration for the CNN\u0026ndash;GRU Model\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParameter\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eValue\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNN Layers\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e3 convolutional\u0026thinsp;+\u0026thinsp;2 max pooling\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFilter Size\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e3\u0026times;3\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGRU Cells\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e128\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLearning Rate\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.001\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBatch Size\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e32\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEpochs\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e50\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFusion Method\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHybrid (Cross-attention\u0026thinsp;+\u0026thinsp;Feature Alignment)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOptimizer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAdam\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eTo assess the performance of the proposed hybrid architecture, the following baseline models were employed:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eCNN-only model: Processes only visual inputs.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eGRU-only model: Processes only textual inputs.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eCNN\u0026ndash;Transformer hybrid model: Combines CNN-extracted visual features with transformer-based text embeddings via an attention mechanism.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eLate Fusion CNN\u0026ndash;GRU\u003c/strong\u003e \u003cp\u003eCNN and GRU predictions are combined after independent training.\u003c/p\u003e \u003c/p\u003e \u003cp\u003ePerformance metrics include accuracy, precision, recall, F1-score, and latency (ms per prediction).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e4.2. Software Simulation of the Proposed Hybrid CNN\u0026ndash;GRU Model\u003c/h2\u003e \u003cp\u003eThe software simulation of the proposed hybrid CNN\u0026ndash;GRU model was implemented to evaluate its real-time performance in processing multimodal data and making synchronized decisions. The simulation was conducted in both Python and MATLAB environments to ensure robustness and reproducibility. In Python, the model was developed using TensorFlow and Keras libraries, where the CNN modules extracted visual features and the GRU layers processed sequential textual inputs. In MATLAB, the Deep Learning Toolbox was employed to configure CNN-based feature extraction and GRU sequence embedding modules. For each environment, the simulation pipeline included the following steps:\u003c/p\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eData Preprocessing - Visual inputs were normalized and augmented using random rotations, scaling, and brightness adjustments to improve generalization. Textual data were tokenized and mapped to 300-dimensional pre-trained GloVe embeddings, which were fed sequentially into the GRU layers.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eFeature Extraction - The CNN layers processed visual data to extract spatial features, while the GRU layers encoded semantic textual information.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eHybrid Fusion - Visual and textual feature vectors were concatenated and integrated through cross-attention and feature alignment mechanisms, forming a unified multimodal representation.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eDecision Layer Simulation - The fused feature vector was passed through fully connected layers followed by a softmax function to generate prediction probabilities for each class.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eReal-Time Performance Evaluation - Latency measurements (in milliseconds per prediction) were recorded alongside prediction outputs to evaluate the feasibility of the model in real-time applications.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003cp\u003eThe simulation results confirmed that the hybrid CNN\u0026ndash;GRU architecture efficiently synchronizes visual and textual modalities, producing accurate predictions with low latency. This setup allows the model to be deployed in practical real-time scenarios such as smart city monitoring, traffic rule violation detection, industrial safety surveillance, and autonomous robotics systems.\u003c/p\u003e \u003cp\u003eThe simulation also enabled comparative testing against baseline models, including single-modal CNN, GRU-only, CNN\u0026ndash;Transformer, and late fusion CNN\u0026ndash;GRU approaches, highlighting the superior performance of the proposed hybrid model in both accuracy and processing efficiency.\u003c/p\u003e \u003cdiv id=\"Sec13\" class=\"Section3\"\u003e \u003ch2\u003e4.2.1. Simulation of a Hybrid CNN\u0026ndash;GRU Model in MATLAB\u003c/h2\u003e \u003cp\u003eFor the simulation of the proposed Hybrid CNN\u0026ndash;GRU model in MATLAB, the German Traffic Sign Recognition Benchmark (GTSRB) dataset was used (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.kaggle.com/datasets/meowmeowmeowmeowmeow/gtsrb-german-traffic-sign\u003c/span\u003e\u003cspan address=\"https://www.kaggle.com/datasets/meowmeowmeowmeowmeow/gtsrb-german-traffic-sign\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e). This dataset is designed for traffic sign recognition and includes 43 different classes (Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e4.1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eDataset link: GTSRB dataset (Kaggle)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eTotal number of images: ~39,000\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eClasses: 43\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eTrain / Validation split: 80% / 20%\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eThe dataset has the following structure on the computer:\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe conducted integral and comparative analysis demonstrates that the Hybrid CNN\u0026ndash;GRU model, implemented in the MATLAB environment, progressively improves its performance during the iterative training process, approaching convergence and ultimately achieving high classification accuracy. The stage-wise evolution of the training dynamics is summarized in the table below \u003cb\u003e(\u003c/b\u003eTable\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e4.2\u003c/span\u003e\u003cb\u003e).\u003c/b\u003e\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab6\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4.2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eIntegral Analysis of Training Dynamics by Stages\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eStage\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEpoch 1\u0026ndash;2 (Initial)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eEpoch 3\u0026ndash;4 (Stabilization)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eEpoch 5\u0026ndash;6 (Convergence)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel behavior\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLearning simple and local features\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLearning intermediate and complex features\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHigh-level representation\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAccuracy (%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e~\u0026thinsp;10\u0026ndash;40%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e~\u0026thinsp;40\u0026ndash;70%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e~\u0026thinsp;70\u0026ndash;95%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLoss\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e~\u0026thinsp;3.5 \u0026rarr; 2.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e~\u0026thinsp;2.1 \u0026rarr; 1.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e~\u0026thinsp;1.3 \u0026rarr; 0.6\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eStability\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eUnstable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eStable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHighly stable\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eValidation alignment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePartial\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHigh\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eStable\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOverfitting risk\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNone\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLow\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eModerate\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eAt the initial stage of training, the model starts with randomly initialized weights, resulting in low accuracy and high loss values. As shown in Table\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e4.2\u003c/span\u003e, the model primarily learns simple and local features during this phase, and the learning dynamics exhibit instability. From the third and fourth epochs onward, the model behavior becomes more stable. The CNN layers extract more complex and informative visual features, while the GRU models the relationships between these features, forming a deeper representation. As indicated in Table\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e4.2\u003c/span\u003e, both accuracy increases and loss decreases more consistently during this stage. In the final stage, the model approaches convergence and learns high-level features. The accuracy reaching the range of 70\u0026ndash;95% indicates that the model can effectively distinguish complex patterns \u003cb\u003e(\u003c/b\u003eTable\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e4.2\u003c/span\u003e\u003cb\u003e).\u003c/b\u003e\u003c/p\u003e \u003cp\u003eThe overall performance metrics of the model are presented in the following Table\u0026nbsp;\u003cspan refid=\"Tab7\" class=\"InternalRef\"\u003e4.3\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab7\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4.3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eModel Performance Metrics\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMetric\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eValue\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eValidation Accuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e94% \u0026ndash; 96%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePrecision\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026asymp;\u0026thinsp;0.95\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026asymp;\u0026thinsp;0.95\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eF1-score\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026asymp;\u0026thinsp;0.95\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFinal Loss\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026asymp;\u0026thinsp;0.6\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNumber of Epochs\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eAs shown in Table\u0026nbsp;\u003cspan refid=\"Tab7\" class=\"InternalRef\"\u003e4.3\u003c/span\u003e and Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e4.2\u003c/span\u003e, the model demonstrates high accuracy and balanced performance. The high values of precision, recall, and F1-score confirm that the model produces reliable and consistent predictions.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eA comparison of the proposed model with other approaches is presented below.As shown in Table\u0026nbsp;\u003cspan refid=\"Tab8\" class=\"InternalRef\"\u003e4.4\u003c/span\u003e, the proposed Hybrid CNN\u0026ndash;GRU model outperforms other approaches in terms of both accuracy and computational efficiency. In particular, the integration of multimodal data enables superior performance compared to unimodal and loosely fused models.In conclusion, the conducted integrated analysis demonstrates that the Hybrid CNN\u0026ndash;GRU model effectively extracts both spatial and sequential features through a multi-stage learning mechanism. The training dynamics (Table\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e4.2\u003c/span\u003e) confirm stable learning behavior, the performance metrics (Table\u0026nbsp;\u003cspan refid=\"Tab7\" class=\"InternalRef\"\u003e4.3\u003c/span\u003e) indicate high accuracy, and the comparative analysis (Table\u0026nbsp;\u003cspan refid=\"Tab8\" class=\"InternalRef\"\u003e4.4\u003c/span\u003e) clearly highlights the superiority of the proposed model. As a result, the model achieves the research objective with approximately 95% accuracy and can be considered an efficient and reliable solution for real-time multimodal artificial intelligence applications.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab8\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4.4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cb\u003eComparative Analysis of Models\u003c/b\u003e\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eF1-score\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eLatency (ms)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eEvaluation\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNN-only\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e91.2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.90\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eVisual-only, limited\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGRU-only\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e85.6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eWeak visual capability\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNN\u0026ndash;Transformer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e93.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eHigh accuracy, computationally heavy\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLate Fusion CNN\u0026ndash;GRU\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e94.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.94\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eModerate performance\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eHybrid CNN\u0026ndash;GRU\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e95.3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e0.95\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e19\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003eMost optimal balance\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section3\"\u003e \u003ch2\u003e4.2.2. Simulation of a Hybrid CNN\u0026ndash;GRU Model in Python (TensorFlow/Keras və PyTorch)\u003c/h2\u003e \u003cp\u003eTo more comprehensively evaluate the functionality and flexibility of the proposed Hybrid CNN\u0026ndash;GRU model, the simulation process was conducted not only in MATLAB but also on the Python platform, specifically using the TensorFlow/Keras and PyTorch libraries. This approach allowed for a comparative analysis of the model's applicability across different software environments, computational efficiency, and compatibility of its learning dynamics.\u003c/p\u003e \u003cp\u003eFor the simulation, the GTSRB dataset, which is based on traffic sign recognition, was utilized. In the preprocessing stage, the visual data were normalized, resized to standard dimensions, and data augmentation techniques (random rotation, scaling, and brightness variations) were applied. The text component was incorporated into the GRU model via a simulated annotation structure and represented by 300-dimensional embedding vectors. In the TensorFlow/Keras environment, the model was built using a higher-level abstraction, where CNN layers were used to extract visual features, and GRU layers were employed to model sequential dependencies. In the PyTorch environment, the same architecture was implemented with lower-level control, allowing for more detailed optimization of the training process. In both environments, the Adam optimizer was used with identical hyperparameters: batch size\u0026thinsp;=\u0026thinsp;32 and learning rate\u0026thinsp;=\u0026thinsp;0.001.\u003c/p\u003e \u003cp\u003eThe comparative results of the training dynamics across different stages are presented in the following Table \u003cspan refid=\"Tab9\" class=\"InternalRef\"\u003e4.5\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab9\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4.5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eTraining dynamics of the Hybrid CNN\u0026ndash;GRU model in Python (TensorFlow vs PyTorch)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eStage\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTensorFlow/Keras\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePyTorch\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eScientific Interpretation\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEpoch 1\u0026ndash;2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy: ~15\u0026ndash;45%, Loss: high\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy: ~10\u0026ndash;40%, Loss: high\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eInitial learning phase\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEpoch 3\u0026ndash;4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy: ~50\u0026ndash;75%, Loss: steadily decreasing\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy: ~45\u0026ndash;70%, Loss: steadily decreasing\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eStabilization phase\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEpoch 5\u0026ndash;6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy: ~85\u0026ndash;96%, Loss: low\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy: ~80\u0026ndash;94%, Loss: low\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eConvergence phase\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLearning stability\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHigh\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMedium\u0026ndash;High\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTensorFlow more stable\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFlexibility\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMedium\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHigh\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePyTorch more flexible\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eAs seen from the table, the learning trajectory of the model is similar in both environments, with a stepwise increase in performance. In TensorFlow/Keras, learning is more stable and converges faster, which can be attributed to the high-level API simplifying the optimization process. In PyTorch, the model allows for more flexible management, although this sometimes requires additional adjustments during training. The overall performance metrics of the model are presented in the following Table\u0026nbsp;\u003cspan refid=\"Tab10\" class=\"InternalRef\"\u003e4.6\u003c/span\u003e and Table\u0026nbsp;\u003cspan refid=\"Tab11\" class=\"InternalRef\"\u003e4.7\u003c/span\u003e\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab10\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4.6\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison of model performance in Python environments\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMetric\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTensorFlow/Keras\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePyTorch\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e95.8%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e94.6%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePrecision\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.96\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.95\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.96\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.94\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eF1-score\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.96\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.94\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLoss\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.55\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.62\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLatency (ms)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e20\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab11\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4.7\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparative Analysis of Models \u0026ndash; Python Results (TensorFlow vs PyTorch)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"8\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy (%) TF\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy (%) PT\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eF1-score TF\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eF1-score PT\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eLatency (ms) TF\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eLatency (ms) PT\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eEvaluation\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNN-only\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e91.2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e90.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.90\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.89\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eVisual-only, limited\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGRU-only\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e85.6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e85.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.83\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eWeak visual capability\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNN\u0026ndash;Transformer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e93.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e93.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.92\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eHigh accuracy, computationally heavy\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLate Fusion CNN\u0026ndash;GRU\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e94.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e94.2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.94\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eModerate performance\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eHybrid CNN\u0026ndash;GRU\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e95.8\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e94.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.96\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e0.94\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e18\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e20\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003eMost optimal balance\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eAs shown in Table\u0026nbsp;\u003cspan refid=\"Tab10\" class=\"InternalRef\"\u003e4.6\u003c/span\u003e, Table\u0026nbsp;\u003cspan refid=\"Tab11\" class=\"InternalRef\"\u003e4.7\u003c/span\u003e and Fig.\u0026nbsp;\u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e4.3\u003c/span\u003e the model demonstrates high accuracy on both platforms. TensorFlow/Keras achieves slightly higher accuracy and lower loss, reflecting the efficiency of its optimization mechanisms. PyTorch shows slightly lower results but provides greater flexibility in model construction, making it more suitable for research-oriented applications.Overall, the comparative analysis demonstrates that the Hybrid CNN\u0026ndash;GRU model exhibits stable and high performance across different software environments. The model successfully approaches convergence on both platforms and effectively integrates multimodal data. While TensorFlow/Keras is more suitable for practical and rapid deployment, PyTorch is preferable for in-depth experimental research. The simulation and comparative analysis conducted in Python confirm that the Hybrid CNN\u0026ndash;GRU model achieves high accuracy and stable learning performance across platforms.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe model achieved approximately 95\u0026ndash;96% accuracy in TensorFlow/Keras and 94\u0026ndash;95% in PyTorch. These results indicate that the proposed approach is both theoretically and practically effective and can be successfully applied in various real-time applications (Table\u0026nbsp;\u003cspan refid=\"Tab11\" class=\"InternalRef\"\u003e4.7\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e4.3. Experimental Results\u003c/h2\u003e \u003cp\u003eThe experimental evaluation of the Hybrid CNN\u0026ndash;GRU model demonstrates its effectiveness in real-time multimodal decision-making tasks, where both visual and textual information are processed synchronously. Using the GTSRB dataset and other publicly available multimodal datasets, the model was tested across MATLAB and Python environments (TensorFlow/Keras and PyTorch), enabling a robust comparative assessment of its learning dynamics, accuracy, and computational efficiency.\u003c/p\u003e \u003cp\u003eIn MATLAB, the Hybrid CNN\u0026ndash;GRU model exhibited a progressive improvement in performance during the iterative training process. During the initial epochs, the model learned simple and local visual features with low accuracy (~\u0026thinsp;10\u0026ndash;40%) and high loss (~\u0026thinsp;3.5 \u0026rarr; 2.1). As training progressed to the stabilization phase, the CNN layers extracted more complex spatial features, while the GRU layers captured sequential dependencies, resulting in improved accuracy (~\u0026thinsp;40\u0026ndash;70%) and steadily decreasing loss (~\u0026thinsp;2.1 \u0026rarr; 1.3). In the convergence phase, the model achieved high-level representation with accuracy reaching\u0026thinsp;~\u0026thinsp;70\u0026ndash;95% and loss decreasing to ~\u0026thinsp;0.6, demonstrating effective generalization. Validation performance closely followed training metrics, confirming the model\u0026rsquo;s stability and reliability.\u003c/p\u003e \u003cp\u003eThe Python simulations provided additional insights into the model's cross-platform performance. TensorFlow/Keras demonstrated slightly higher accuracy (~\u0026thinsp;95.8%) and lower loss (0.55) compared to PyTorch (~\u0026thinsp;94.6% accuracy, 0.62 loss), reflecting the efficiency of high-level API optimization. PyTorch, however, offered greater flexibility for detailed model control and experimentation. Across both platforms, the training dynamics showed consistent improvement, with a smooth reduction of loss and stepwise increase in accuracy, confirming stable convergence.\u003c/p\u003e \u003cp\u003eA comparative analysis against baseline models further highlights the superiority of the proposed Hybrid CNN\u0026ndash;GRU model. The CNN-only model, limited to visual data, achieved 91\u0026ndash;92% accuracy, while the GRU-only model, constrained to textual sequences, reached 85\u0026ndash;86% accuracy. CNN\u0026ndash;Transformer hybrids provided high accuracy (93\u0026ndash;94%) but incurred heavier computational costs. Late fusion CNN\u0026ndash;GRU approaches offered moderate performance (~\u0026thinsp;94%). The proposed Hybrid CNN\u0026ndash;GRU consistently outperformed all baselines, achieving the highest accuracy, F1-score, and low latency (~\u0026thinsp;18\u0026ndash;20 ms per prediction), indicating its suitability for real-time applications.\u003c/p\u003e \u003cp\u003eThe model\u0026rsquo;s overall performance metrics summarize its balanced and robust characteristics:\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab12\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4.8\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eOverall performance metrics\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMetric\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMATLAB\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTensorFlow/Keras\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePyTorch\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAccuracy (%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e94\u0026ndash;96\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e95.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e94.6\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePrecision\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.95\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.96\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.95\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.95\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.96\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.94\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eF1-score\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.95\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.96\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.94\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLoss\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.55\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.62\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLatency (ms)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e20\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eIn addition, the comparative evaluation of different models in Python is presented as follows:\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Taba\" border=\"1\"\u003e \u003ccolgroup cols=\"8\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy (%) TF\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy (%) PT\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eF1-score TF\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eF1-score PT\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eLatency (ms) TF\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eLatency (ms) PT\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eEvaluation\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNN-only\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e91.2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e90.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.90\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.89\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eVisual-only, limited\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGRU-only\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e85.6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e85.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.83\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eWeak visual capability\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNN\u0026ndash;Transformer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e93.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e93.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.92\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eHigh accuracy, computationally heavy\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLate Fusion CNN\u0026ndash;GRU\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e94.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e94.2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.94\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eModerate performance\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eHybrid CNN\u0026ndash;GRU\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e95.8\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e94.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.96\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e0.94\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e18\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e20\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003eMost optimal balance\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe analysis demonstrates that the Hybrid CNN\u0026ndash;GRU model successfully integrates spatial and sequential features through cross-attention and feature alignment mechanisms. Training dynamics confirm stable learning, while performance metrics indicate high accuracy and efficiency. Its comparative advantage over unimodal, CNN\u0026ndash;Transformer, and late fusion models is evident, particularly in the combined achievement of accuracy, low latency, and robust F1-score.\u003c/p\u003e \u003cp\u003eOverall, the experimental evaluation confirms that the Hybrid CNN\u0026ndash;GRU model is highly effective, platform-independent, and suitable for real-time multimodal AI applications, including traffic monitoring, autonomous systems, and industrial or smart city surveillance. TensorFlow/Keras offers a faster and more stable practical deployment, while PyTorch provides flexibility for research-driven experimentation. The model achieves approximately 95\u0026ndash;96% accuracy in real-time scenarios, validating both its theoretical design and practical implementation.\u003c/p\u003e \u003c/div\u003e"},{"header":"5. Conclusion","content":"\u003cp\u003eThe experimental evaluation of the Hybrid CNN\u0026ndash;GRU model, implemented in both Python (TensorFlow/Keras and PyTorch) and MATLAB (Deep Learning Toolbox), confirms its effectiveness for real-time multimodal decision-making through the synchronized processing of visual and textual inputs. Across all conducted simulations, the model consistently outperformed single-modal approaches, including CNN-only and GRU-only models, as well as late fusion and CNN\u0026ndash;Transformer architectures. In Python, the Hybrid CNN\u0026ndash;GRU achieved an accuracy of 95\u0026ndash;96% in TensorFlow/Keras and 94\u0026ndash;95% in PyTorch, with F1-scores of 0.96 and 0.94, respectively, while maintaining low latency (18\u0026ndash;20 ms per prediction), confirming its suitability for real-time applications. MATLAB simulations similarly demonstrated stable convergence and high performance, with validation accuracy ranging from 94% to 96% and a final loss of approximately 0.6.\u003c/p\u003e \u003cp\u003eThe superior performance of the proposed model is attributed to its hybrid architecture, which effectively extracts spatial features from visual inputs via CNN layers and models sequential dependencies in textual data using GRU layers. The cross-attention and feature alignment mechanisms enable seamless integration of multimodal information, allowing the model to capture nuanced patterns such as traffic rule violations more reliably than unimodal or post-fusion methods. Comparative analysis shows that the Hybrid CNN\u0026ndash;GRU model achieves the most optimal balance between predictive accuracy, computational efficiency, and robustness, outperforming alternative architectures in both accuracy and latency.\u003c/p\u003e \u003cp\u003eFurthermore, SHAP-based interpretability analysis in Python confirmed that the model transparently exploits interactions between visual and textual modalities, enabling explainable predictions and supporting trustworthiness in safety-critical real-time systems. The flexibility of the model in PyTorch allows for research-oriented experimentation and detailed optimization, whereas TensorFlow/Keras provides faster convergence and practical deployment advantages.\u003c/p\u003e \u003cp\u003eIn conclusion, the Hybrid CNN\u0026ndash;GRU framework, validated across multiple software platforms, demonstrates a robust, scalable, and interpretable solution for real-time multimodal AI applications. Its consistent high accuracy, low latency, and stable learning behavior make it particularly suitable for smart city management, traffic surveillance, industrial safety monitoring, and autonomous robotic systems, offering both theoretical and practical efficiency in diverse real-time operational scenarios.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e \u003ch2\u003eConsent to Publish:\u003c/h2\u003e \u003cp\u003eNot applicable.\u003c/p\u003e \u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eA.M. (Aida Mustafayeva) conceptualized and designed the study, supervised the research, and revised the manuscript.E.I. (Elmira Israfilova) developed the hybrid CNN\u0026ndash;GRU model, performed experiments, and analyzed the results.G.B. (Gunel Baxshiyeva) prepared the figures, tables, and data visualization.S.A. (Saadat Aslanova) contributed to data preprocessing, simulation, and manuscript drafting.All authors reviewed and approved the final version of the manuscript.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eData Availability Statement (Optimal Version):Yes. The datasets used and/or analyzed during the current study are publicly available. The traffic image dataset can be accessed at https://www.kaggle.com/datasets/meowmeowmeowmeowmeow/gtsrb-german-traffic-sign. Any additional data supporting the findings of this study are available from the corresponding author upon reasonable request.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eAntol S, Agrawal A, Lu J, Mitchell M, Batra D, Zitnick L, C., Parikh D. (2015). VQA: Visual Question Answering. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2425\u0026ndash;2433. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/ICCV.2015.279\u003c/span\u003e\u003cspan address=\"10.1109/ICCV.2015.279\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAcosta JN, Falcone GJ, Rajpurkar P, Topol EJ. Multimodal biomedical artificial intelligence. Nat Med. 2022;28:1773\u0026ndash;84. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41591-022-01981-2\u003c/span\u003e\u003cspan address=\"10.1038/s41591-022-01981-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBinte Rashid M, Rahaman MS, Rivas P. (2024). Navigating the Multimodal Landscape: A Review on Integration of Text and Image Data. Machine Learning and Knowledge Extraction. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/make6030074\u003c/span\u003e\u003cspan address=\"10.3390/make6030074\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen X, Xie H, Tao X, Wang FL, Leng M, Lei B. (2024). \u003cem\u003eArtificial intelligence and multimodal data fusion for smart healthcare: topic modeling and bibliometrics.\u003c/em\u003e Artificial Intelligence Review (Springer). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10462-024-10712-7\u003c/span\u003e\u003cspan address=\"10.1007/s10462-024-10712-7\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDixit C, Satapathy SM. Deep CNN with late fusion for real-time multimodal emotion recognition. Expert Syst Appl. 2024;240., Article 122579. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.eswa.2023.122579\u003c/span\u003e\u003cspan address=\"10.1016/j.eswa.2023.122579\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHao X, Du H, Guo J et al. (2025). A CNN\u0026ndash;Transformer Hybrid Model for Multimodal Person Re-Identification. International Journal of Multimedia Information Retrieval. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s13735-025-00367-7\u003c/span\u003e\u003cspan address=\"10.1007/s13735-025-00367-7\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang M, Jia S, Chang M-C, Lyu S. Text-image de-contextualization detection using vision-language models. In \u003cem\u003eProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2022)\u003c/em\u003e, Virtual, 7\u0026ndash;13 May 2022.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGupta C, Gill NS, Gulia P et al. (2025). \u003cem\u003eA multimodal fusion model for real-time emotion recognition using audio-visual-textual features.\u003c/em\u003e Journal of Big Data (Springer). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s40537-025-01300-9\u003c/span\u003e\u003cspan address=\"10.1186/s40537-025-01300-9\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu Y, Zhu X, Clifton DA. Multimodal Learning with Transformers: A Survey. IEEE Trans Pattern Anal Mach. 2023. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/TPAMI.2023.3275156\u003c/span\u003e\u003cspan address=\"10.1109/TPAMI.2023.3275156\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Intelligence (TPAMI).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi G, Ren G, Wang J, Yu Z, Jiang B, Guo Q. (2025). \u003cem\u003eMultimodal fusion transformer network for multispectral pedestrian detection in low-light condition.\u003c/em\u003e Scientific Reports (Nature).\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41598-025-03567-7\u003c/span\u003e\u003cspan address=\"10.1038/s41598-025-03567-7\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu Y. (2024). Multimodal NLP and Cross-Media Information Understanding. Proceedings of SDMC 2024. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2991/978-2-38476-327-6_24\u003c/span\u003e\u003cspan address=\"10.2991/978-2-38476-327-6_24\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMakhmudov F, Kultimuratov A, Cho Y. Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures. Appl Sci. 2024;14(10):4199. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/app14104199\u003c/span\u003e\u003cspan address=\"10.3390/app14104199\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMeel P, Vishwakarma DK. Multi-modal fusion using fine-tuned self-attention and transfer learning for veracity analysis of web information. Expert Syst Appl. 2023;229:120537.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNakach F-Z, Idri A, Goceri E. A comprehensive investigation of multimodal deep learning fusion strategies for breast cancer classification. Artif Intell Rev Springer DOI. 2024. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10462-024-10984-z\u003c/span\u003e\u003cspan address=\"10.1007/s10462-024-10984-z\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRasheed J, Jamil A, Hasibe B. Turkish Text Detection System from Videos Using Machine Learning and Deep Learning Techniques. IEEE Third International Conference on Data Stream Mining \u0026amp; Processing August 21\u0026ndash;25, 2020, Lviv, Ukraine. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/DSMP47368.2020.9204036\u003c/span\u003e\u003cspan address=\"10.1109/DSMP47368.2020.9204036\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShi D, Zhang W, Yang J et al. (2025).A multimodal vision\u0026ndash;language foundation model for computational medicine. npj Digital Medicine. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41746-025-01772-2\u003c/span\u003e\u003cspan address=\"10.1038/s41746-025-01772-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShaikh MB, Islam SMS, Chai D, Akhtar N. Multimodal fusion for audio-image and video action recognition. Neural Comput Appl. 2024;Q1. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s00521-023-09186-5\u003c/span\u003e\u003cspan address=\"10.1007/s00521-023-09186-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShao W, Fan D, Cui C et al. (2026). Deep learning-based astronomical multimodal data fusion. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.inffus.2025.104103\u003c/span\u003e\u003cspan address=\"10.1016/j.inffus.2025.104103\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTsai YHH, Bai S, Yamada M, Morency LP, Salakhutdinov R. (2019). Multimodal Transformer for Multimodal Sentiment Analysis. Proceedings of the ACL. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.18653/v1/P19-1623\u003c/span\u003e\u003cspan address=\"10.18653/v1/P19-1623\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang J-H, Norouzi M, Tsai SM. Augmenting Multimodal Content Representation with Transformers for Misinformation Detection. Big Data Cogn Comput. 2024;8(10):134. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/bdcc8100134\u003c/span\u003e\u003cspan address=\"10.3390/bdcc8100134\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang H. (2024). Multimodal Audio-Visual Fusion Using 3D CNN and CRNN for Behavior Recognition. Frontiers in Neurorobotics. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fnbot.2024.1284175\u003c/span\u003e\u003cspan address=\"10.3389/fnbot.2024.1284175\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eXu P, Zhu X, Clifton DA. Multimodal Learning with Transformers: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/TPAMI.2023.3275156\u003c/span\u003e\u003cspan address=\"10.1109/TPAMI.2023.3275156\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVinyals O, Toshev A, Bengio S, Erhan D. Show and tell: A neural image caption generator. In \u003cem\u003eProceedings of the IEEE Conference on Computer Vision and Pattern Recognition\u003c/em\u003e, Boston, MA, USA, 7\u0026ndash;12 June 2015; pp. 3156\u0026ndash;3164.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhao Y, Mamat M, Aysa A et al. (2023). Multimodal Sentiment System Based on CRNN-SVM. Neural Computing and Applications. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s00521-023-08366-7\u003c/span\u003e\u003cspan address=\"10.1007/s00521-023-08366-7\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZengyi Yang Y, Li X, Tang et al. (2024). \u003cem\u003eMGFusion: A multimodal large language model-guided framework for image fusion.\u003c/em\u003e Frontiers in Neurorobotics. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fnbot.2024.1521603\u003c/span\u003e\u003cspan address=\"10.3389/fnbot.2024.1521603\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang D, Wong WK, Chew IM. (2025). \u003cem\u003eA comprehensive review of multimodal visual representation learning: tracing the evolution from CNNs to transformers and beyond.\u003c/em\u003eInternational Journal of Multimedia Information Retrieval (Springer). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s13735-025-00382-8\u003c/span\u003e\u003cspan address=\"10.1007/s13735-025-00382-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang Z, Li Y, Tang X, Xie M. MGFusion: A multimodal large language model-guided framework for image fusion. Front Neurorobotics. 2024;18. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fnbot.2024.1521603\u003c/span\u003e\u003cspan address=\"10.3389/fnbot.2024.1521603\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"discover-artificial-intelligence","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"diai","sideBox":"Learn more about [Discover Artificial Intelligence](https://www.springer.com/44163)","snPcode":"","submissionUrl":"","title":"Discover Artificial Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Hybrid CNN–GRU Neural Networks, Multimodal Decision-Making, Image and Text Analysis, Multimodal Data Processing, Deep Leraning","lastPublishedDoi":"10.21203/rs.3.rs-9257523/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9257523/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis study presents a hybrid CNN\u0026ndash;GRU model for the synchronous processing of visual and textual information, designed to support real-time multimodal decision-making. The proposed architecture integrates CNN-based visual feature extraction with GRU-based sequential text processing, while cross-attention and feature alignment mechanisms enable effective fusion of the two modalities. This approach represents a significant advancement over conventional unimodal and late-fusion methods, as it allows real-time, synchronized multimodal integration rather than post-hoc combination of separate predictions. Unlike CNN\u0026ndash;Transformer architectures, the model achieves high predictive performance with lower computational cost and reduced latency, making it more suitable for practical real-time applications. Evaluations in Python (TensorFlow/Keras and PyTorch) and MATLAB demonstrate that the Hybrid CNN\u0026ndash;GRU model achieves high accuracy (95\u0026ndash;96% in TensorFlow/Keras, 94\u0026ndash;95% in PyTorch), precision (0.96 / 0.95), recall (0.96 / 0.94), and F1-score (0.96 / 0.94), while maintaining low computational latency (18\u0026ndash;20 ms per prediction). SHAP-based interpretability analysis confirms that the model effectively exploits interactions between visual and textual modalities, providing transparent and explainable predictions. Overall, the Hybrid CNN\u0026ndash;GRU framework offers an optimal combination of high predictive performance, computational efficiency, interpretability, and real-time applicability, making it suitable for smart city management, traffic monitoring, industrial safety, and autonomous robotic systems.\u003c/p\u003e","manuscriptTitle":"Hybrid Cnn-gru Model for Real-time Multimodal Decision-making in Image and Text Analysis","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-24 07:18:34","doi":"10.21203/rs.3.rs-9257523/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-05-13T07:06:03+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-25T05:06:15+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-23T13:37:51+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"115230247092738889860295426758630651556","date":"2026-04-20T09:18:49+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"242772348581790032112392920273491387548","date":"2026-04-20T03:42:54+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"132257185480058340854788560186978355677","date":"2026-04-17T16:52:41+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-17T05:46:21+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"203640272969719497011784663607464435390","date":"2026-04-17T05:20:36+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"223386121101907473099696454427189982988","date":"2026-04-17T05:18:40+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"165002353941815708736100671630482445271","date":"2026-04-17T04:58:44+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-17T04:40:09+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-04-01T01:38:17+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-04-01T01:37:38+00:00","index":"","fulltext":""},{"type":"submitted","content":"Discover Artificial Intelligence","date":"2026-03-29T08:40:53+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"discover-artificial-intelligence","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"diai","sideBox":"Learn more about [Discover Artificial Intelligence](https://www.springer.com/44163)","snPcode":"","submissionUrl":"","title":"Discover Artificial Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"4c4bbb60-8f51-47b6-a1b3-1b3e03b688ce","owner":[],"postedDate":"April 24th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Revision requested","date":"2026-05-13T07:06:03+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[],"tags":[],"updatedAt":"2026-05-13T07:11:27+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-24 07:18:34","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9257523","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9257523","identity":"rs-9257523","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}