A Multimodal Approach to Knowledge Tracing for Personalized Learning based on Cognitive and Affective Data in Resource-Constrained Environments

A Multimodal Approach to Knowledge Tracing for Personalized Learning based on Cognitive and Affective Data in Resource-Constrained Environments | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Multimodal Approach to Knowledge Tracing for Personalized Learning based on Cognitive and Affective Data in Resource-Constrained Environments Wilson Ng'andu Mwiiya, Aaron Zimba This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9232899/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 6 You are reading this latest preprint version Abstract Personalized learning remains a critical challenge in education, particularly in resource-constrained environments such as Zambia, where instructional practices often follow a one-size-fits-all approach that overlooks differences in learners’ cognitive processes, behaviours, and emotional states. In STEAM education, this limitation contributes to learner disengagement and poor academic performance. Although knowledge tracing (KT) techniques provide data-driven methods for modeling student learning, most existing approaches focus primarily on cognitive outcomes and fail to capture the multimodal nature of the learning process. This study proposes a multimodal approach to knowledge tracing for personalized learning that integrates cognitive and affective data to provide a more comprehensive representation of learner behaviour. The proposed framework incorporates five modalities: knowledge mastery estimates, programming interaction features, textual help-seeking patterns, affective states, and behavioural indicators. These heterogeneous data sources are combined using a context-aware deep learning architecture with hierarchical fusion and adaptive gating, enabling dynamic weighting of modalities and robustness to missing data-an important requirement in resource-constrained settings. The model was evaluated on three benchmark datasets (ASSISTments 2015, CSEDM 2019, and XES3G5M) using five-fold cross-validation on 1,065 student interaction sequences. The results demonstrate strong predictive performance, achieving a mean AUC-ROC of 0.9933 ± 0.0006, substantially outperforming traditional Bayesian Knowledge Tracing and standard Deep Knowledge Tracing models. Analysis of modality contributions shows that affective states (43.7%) and behavioural indicators (26.3%) are the most influential predictors of learning outcomes, providing empirical evidence of the importance of non-cognitive factors in personalized learning. These findings highlight the value of multimodal knowledge tracing in enhancing personalization and improving learner modelling, particularly in environments with limited educational resources. The proposed approach offers a scalable and context-aware solution for intelligent tutoring systems aimed at supporting inclusive and effective STEAM education in higher education contexts. Multimodal knowledge tracing personalized learning affective computing in education learning analytics resource-constrained educational environments Figures Figure 3 1 INTRODUCTION 1.1 Background and Educational Context Personalized learning has become a central objective in higher education, particularly in Science, Technology, Engineering, Arts, and Mathematics (STEAM) disciplines where learners exhibit diverse cognitive abilities, learning behaviours, and emotional responses. STEAM education plays a critical role in developing problem-solving skills and innovation capacity required for addressing real-world challenges [1] - [3]. However, in resource-constrained environments such as Zambia, educational systems face persistent challenges including limited teaching capacity, high learner-to-instructor ratios, and inadequate infrastructure [4]. These constraints often result in the adoption of one-size-fits-all instructional approaches that do not adequately account for individual learner differences, including variations in learning styles and affective states [5], [6]. Consequently, many students experience disengagement, poor academic performance, and reduced retention in STEAM programmes, highlighting the need for scalable and adaptive learning solutions. Knowledge tracing (KT) has emerged as a key technique in learning analytics for modelling students’ evolving knowledge states based on interaction data. Traditional approaches such as Bayesian Knowledge Tracing (BKT) [7] and Deep Knowledge Tracing (DKT) [9] have demonstrated success in predicting student performance; however, they predominantly rely on correctness-based cognitive signals and overlook the broader multimodal nature of the learning process. Educational research shows that learning outcomes are influenced not only by cognitive factors but also by affective states and behavioural patterns [19] - [21], while multimodal learning analytics emphasizes the integration of diverse data sources to capture complex learning dynamics [17], [18]. Despite these advances, existing multimodal approaches often employ static or simplistic fusion strategies that fail to capture the dynamic and context-dependent relationships among modalities. To address this gap, this study proposes a multimodal approach to knowledge tracing that integrates cognitive and affective data through a context-aware deep learning framework, enabling more accurate and adaptive modelling of learner behaviour in resource-constrained educational environments. 1.2 Knowledge Tracing and Personalized Learning Knowledge tracing represents a data driven approach to modelling the evolving knowledge states of students based on interaction sequences. Traditional KT approaches like Bayesian Knowledge Tracing (BKT), inspired by the concept of mastery learning, model student knowledge as a binary latent variable [7] hidden from direct observation and usually achieve an AUC-ROC score of between 0.65 and 0.75 on benchmark datasets The limitations of foundational BKT models, success in deep learning [8] and the need to model more complex patterns inspired the development of Deep Knowledge Tracing (DKT) [9]. DKT makes use of recurrent neural networks to capture complex temporal dependencies in learning sequencies and has an improved AUC-ROC score ranging from 0.80-0.86. Despite these advances in data driven solutions [10], [11], [12], [13], [14], [15], [16], the notable foundational Knowledge Tracing models rely on simple correctness sequences and overlook the rich multimodal nature of the learning process. According to [17], the complex process of learning, especially in the STEAM domain involves problem solving, collaboration, experimentation which produce information that cannot simply be captured by correctness alone [18]. Educational psychology research has it that learning outcomes are not only influenced by cognitive factors but also affective states of learners [19], [20], [21]. This limitation led to the emergence of multimodal learning analytics, which argues that the fusion of diverse data streams including behavioural patterns, textual responses, domain-specific content, and emotional states present a holistic view of the learning process. This study makes the following contributions: 1. A multimodal knowledge tracing framework that integrates cognitive, behavioural, and affective data to model learner knowledge states more comprehensively than traditional cognition-focused approaches. 2. A context-aware hierarchical fusion mechanism with adaptive gating that dynamically learns the relative importance of multiple modalities, enabling robust performance even when some data sources are missing-an important requirement for resource-constrained environments. 3. Empirical evidence on the role of affective and behavioural factors in learning, demonstrating that non-cognitive signals contribute more significantly to performance prediction than traditional correctness-based indicators. 4. A scalable and generalisable approach validated across multiple educational datasets, showing strong predictive performance and applicability across different learning domains, including mathematics and programming. The remainder of this paper is structured as follows. Section 2 reviews related work on multimodal learning analytics and knowledge tracing. Section 3 presents the proposed methodology and model architecture. Section 4 reports the experimental results and analysis. Section 5 discusses the findings and implications, and Section 6 concludes the paper with directions for future research. 2 RELATED WORK 2.1 Multimodal Fusion Mechanisms in Learning Systems The fusion of multimodal data in deep learning models has become a central problem in artificial intelligence research. Traditional approaches to multimodal fusion techniques in education fall in three broad categories based on the point in the pipeline where integration happens: early (feature level), late (decision level) and intermediate (hybrid) fusion [ 22 ]. Early fusion approaches [ 22 ], [ 23 ] assume homogeneous feature representations, which is rarely valid in educational data characterized by heterogeneous modalities such as affective states and behavioural signals. Late fusion methods [ 22 ], [ 24 ] preserve modality-specific information but fail to capture cross-modal dependencies critical for modelling learning processes. Intermediate fusion strategies, including attention-based models [ 25 ], provide improved integration but often lack hierarchical structure and contextual adaptation. For example,[ 26 ] proposed a hierarchical transformer used for classification of documents, however, the fusion strategy that was used was not optimized for temporal educational data. And in a similar manner, [ 27 ] models modalities as heterogeneous graphs, however, the study does not incorporate contextual gating for adaptive fusion. According to [ 28 ], intermediate fusion outperforms early and late fusion in a number of learning scenarios as they are able to capture additional cross modal information while maintaining their individual characteristics. The field has however, undergone a notable shift by focusing on more implicit approaches that modulate importance of each modality based on context and demand. A number of recent research studies have refined the intermediate fusion into a number of categories reflecting modern architectural choices [ 29 ], [ 30 ], [ 31 ], [ 32 ]. The single level fusion combines modalities at a specific network layer and the hierarchical fusion integrates modalities across multiple depths in a network while the attention mechanism assigns dynamic learned weights to different modalities. However, these approaches rarely integrate multimodal signals into knowledge tracing frameworks. 2.2 Gating Mechanisms and Context-Aware Modelling The gated fusion mechanism, a family of neural network architectures that makes use of multiplicative gates to control how information from multiple sources or modalities is combined has been used to dynamically weigh modalities [ 29 ]. The adaptive weighting makes it possible for the model to prioritize informative modalities. An example of this idea was proposed by [ 33 ] as the Gated Multimodal Unit (GMU), laying foundation for gated multimodal fusion. Mostly, these techniques rely on modality specific features and fail to condition the weights on external contextual signals like time elapsed or difficult level. A hierarchical aggregation for federated learning in educational data is introduced by [ 34 ] but does not address context adaptation. Other studies like [ 25 ], [ 35 ] have made use of cross modal attention, however, the gating strategy used is neither hierarchical nor sensitive to context. Context aware fusion builds on adaptive gating [ 36 ] by allowing the fusion process to respond to contextual information and the Context-Aware Attentive Knowledge Tracing (AKT) model by [ 18 ] highlights the benefits of including context in educational knowledge tracing. As a result the model showed a 6% improvement in AUC on standard benchmark datasets and maintained interpretability through clearly defined attention weights. Existing approaches do not simultaneously model hierarchical fusion, context-awareness, and multimodal knowledge tracing, particularly in resource-constrained educational environments. Despite significant advances in multimodal learning analytics, existing approaches exhibit three key limitations. First, most knowledge tracing models remain predominantly cognitive, neglecting the role of affective and behavioural factors in learning. Second, current multimodal fusion techniques often rely on static or shallow integration strategies that fail to capture complex and hierarchical relationships among modalities. Third, limited attention has been given to context-aware adaptive mechanisms that can dynamically adjust modality importance, particularly in resource-constrained educational settings where data availability may be inconsistent. These limitations highlight the need for a unified framework that integrates multimodal data through context-aware and hierarchical fusion to support more accurate and scalable personalized learning. 3 METHODS Three benchmark datasets were used in the study. These datasets include ASSISTments2015, CSEDM19 and XES3G5M. The ASSISTments2015 dataset retrieved from ASSISTments skill builder problem sets, has detailed records of student interactions with an intelligent tutoring system, which includes student responses to class assessment activities, problem solving attempts, and performance data. The dataset contains roughly 708,631 interactions from over 19,917 students and 100 skills [ 37 ] providing a robust foundation for training and evaluating DKT models. A code specific datasets CSEDM19 was used to test the ability of the framework in handling programming data. This dataset from the CSEDM Challenge provided code submissions from programming tasks and is ideal for knowledge tracing. One mathematics specific datasets XES3G5M was used to demonstrate generalizability of the model beyond ASSISTments. The XES3G5M [ 38 ] dataset is a large public benchmark dataset that contains 5.5 million student interactions with 7,652 math questions and 865 knowledge components. The dataset has rich contextual information which includes text question, solution analysis and a tree structured knowledge component relation. Two types of question are used, multiple choice (1,510 questions) and fill in the blank (6,142 questions) 3.1 Context-Aware Hierarchical Fusion with Learned Gating Architecture 3.1.1 Modality Encoding and Representation Learning The CAHF-LG model takes in five distinct modalities as input, each modality captures different aspects of the student learning behaviour. These modalities are processed using dedicated encoder networks projecting these heterogeneous inputs to a unified 128-dimensional latent space enabling cross modal attention and fusion operations. Each of the modality encoders is made up of a two-layer neural network that uses layer normalization with GELU activation functions. To represent the encoding modality mathematically, $$\:{h}_{m}=\text{LayerNorm}\left(\text{GELU}\left(\text{LayerNorm}\left({W}_{2}\cdot\:\text{GELU}\left({W}_{1}\cdot\:{x}_{m}+{b}_{1}\right)+{b}_{2}\right)\right)\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\right(1)$$ where W₁ has dimensions 128 × dₘ, W₂ is 128 × 128, and xₘ is just the raw input. This setup projects each modality into a common representational space making it possible for comparisons and interactions cross modalities and preserves modality unique characteristics through learned non linear transformations Table 1 Input Modalities and Dimensions Modality Dimension Description Math 1 Knowledge estimate from BKT (Ln) or running accuracy Code 1 Programming event type score (Submit, Compile, Edit) Text 1 Hint and scaffold usage patterns Affect 6 Affective states: Bored, Concentrating, Confused, Frustrated, Off-task, Gaming Behavioral 3 Help-seeking, attempt persistence, time management 3.1.2 Context Encoding Mechanism The context encoder generates a detailed picture of where the learner stands and this picture, through the gating mechanisms dynamically adjusts how information flows. The Context Encoder does not treat the learner’s context as static metadata but dynamically constructs rich context representation that includes the context features and the current affective state of the learner. This approach creates a rich context vector that is then used to condition the gating mechanisms at all the three fusion levels. This in effect creates an adaptive multimodal processing. The base context vector is made of four components, 1) problem type encoding (0.0 for math, 0.5 for mixed, 1.0 for code), 2) student proficiency estimated as the running average of correctness, 3) learning phase indicating position in the sequence normalized to [0,1], and 4) task difficulty estimated as the inverse of student performance. these four scalar values are the concatenated with the affect context embedding resulting from the computation of the mean affective state across the sequence. This operation produces a 128-dimensional context vector capturing both the stable learner characteristics and the dynamic emotional state 3.1.3 Three-Level Hierarchical Fusion Architecture The main architectural contribution of the CAHF-LG is in its three-level hierarchical fusion design. The multimodal information is processed at three distinct granularities by the hierarchical fusion mechanism. This gradually integrates features from the individual modalities to temporal sequences. Each level includes context-aware gating to make sure that the fusion adapts to the specific learner context at every stage of processing. The first fusion level is the feature level fusion which operates on each of the modalities independently and applies self-attention to capture intra model dependencies. For a given modality m , the feature matric that is encoded \(\:{h}_{m}\in\:{\mathbb{ℝ}}^{T\times\:128}\:\) is first adjusted through a context driven gating mechanism where the gate: $$\:{g}_{m}=\sigma\:\left({W}_{g}\cdot\:\text{context}\right)\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\left(2\right)$$ where σ is the sigmoid activation that regulates feature importance. These gated features then go through a multi-head self-attention which is later on followed by a position-wise feedforward network. This fusion level helps the model to identify the most relevant features within each modality given the current learner context. The second level of fusion involves integrating information from all the five modalities. To be able to learn how important each modality is given a particular context, an adaptive gating mechanism was used. The concatenation of pooled modalities and the context vector are fed as input to the adaptive gate network which produces an output of five-dimensional weight vector: with f as the three layer multi layer perceptron MLP, base_weight being the learned prior weights, and τ is a trainable temperature parameter. These weights are used to compute the weighted sum of the modality representations, resulting into a fused multimodal embedding. In addition, before this, aggregation, a cross-modal attention with four heads is applied, as it allows modalities to exchange information before the final weighted combination. The third fusion level is the temporal level fusion which models the temporal dependencies across the learning sequencies by making use of a Transformer encoder with causal masking. This fusion level has a temporal fusion module that is made of four transformer layers with each layer having 8 attention heads and processing sequencies of up to 200 timesteps. To make sure that predictions at a particular timestep t only attends to information from timesteps 1 to t-1, causal masking is used. This approach maintains the proper autoregressive structure needed for knowledge tracing. Then finally Learnable positional encodings are added to capture sequential ordering information 3.1.4 Adaptive Gating Mechanism The adaptive gating mechanism is the main component that enables context-aware multimodal fusion. Instead of relying on a fixed attention mechanism, the adaptive gate module learns to produce dynamic weights that would adapt based on both input features and leaner context. The gating network architecture is made of three connected layers with GELU activations, gradually reducing dimensionality of input until it matches that of the items being gated Linear(context_dim + hidden_dim → hidden_dim) → GELU → Dropout → Linear(hidden_dim → hidden_dim/2) → GELU → Linear(hidden_dim/2 → num_items) The output is merged with learnable base weights encoding prior assumptions about the relevance of modalities and then adjusting them using a learnable temperature parameter before applying SoftMax normalization. These learnable base weights act as a form of regularisation, pushing the gating mechanism towards the preferred modality configurations when the input signal is not strong. During training, the learnable base weights are updated to capture the average contribution from each modality across the datasets. Low values encourage the model to focus on the few modalities and large values lead to a more uniform distribution. Because the temperature is optimized with the rest of the model parameters, it makes it possible for the system to automatically find an appropriate level of balance between specialization and flexibility for a task given Table 2 Three-Level Hierarchical Fusion Specifications Fusion Level Mechanism Purpose Level 1: Feature Context-gated self-attention (4 heads) + FFN per modality Capture intra-modal dependencies; context-aware feature importance Level 2: Modality Adaptive gating + cross-modal attention (4 heads) Learn modality importance weights; enable cross-modal information flow Level 3: Temporal Transformer encoder (4 layers, 8 heads) with causal mask Model temporal evolution of knowledge state across learning sequence 4 RESULTS ANALYSIS 4.1 Overall Model Performance The framework achieved exceptional predictive performance across all evaluation metrics. On the validation set, the model achieved an AUC-ROC of 0.9944 demonstrating its exceptional predictive performance. This result exceeds the performance of traditional knowledge tracing approaches by showing a near perfect discrimination between correct and incorrect student responses. The accuracy of the model was at 95.66% meaning the model could correctly predict student response outcome 95 out of every 100 interactions. An F1 score of 0.9619 was recorded confirming that the accuracy was balanced across the correct and incorrect response classes while the model maintained strong precision and recall. Additional insight into the quality of the predictions was obtained from the Root Mean Square Error (RMSE) of 0.1732. Such a low RMSE means that if the model predicts a correctness of 70% probability, then the actual observed correctness rate is close to 70%. Table 3 summarizes the primary performance metrics achieved by the MDKT CAHF-LG framework. Table 3 CAHF-LG Overall Performance Metrics Metric Value AUC-ROC 0.9944 Accuracy 95.66% F1 Score 0.9619 RMSE 0.1732 Precision (Correct Class) 0.96 Recall (Correct Class) 0.97 4.2 Per-Dataset Performance The performance of the CAHF-LG framework was also evaluated separately across the three structurally different datasets, each with distinct context and characteristics. This per dataset evaluation reveals how the unified multimodal architecture adapts to different educational domains and data availability patterns. The highest performance was achieved by the XES3G5M dataset with an AUC of 0.9986 and accuracy of 98.10%. This reflects the availability of large volume of training data and a consistent response pattern. The ASSISTments dataset had an AUC of 0.9878 and accuracy of 93.74% showing strong performance on the dataset with the richest features including affect state. Comparatively lower AUC performance of 0.8957 and accuracy of 92.31% was recorded from the CSEDM dataset. The low performance on this dataset is due to the small sample size with on 68 students and the variable nature of problem solving in programming tasks compared to mathematics Table 4 Per-Dataset Performance Comparison Dataset AUC-ROC Accuracy PR-AUC Samples ASSISTments 0.9878 93.74% 0.9812 19,707 CSEDM 0.8957 92.31% 0.7734 533 XES3G5M 0.9986 98.10% 0.9997 16,134 4.3 Confusion Matrix Analysis The model correctly classified 34,794 instances (95.66% accuracy) out of a total of 36,374 predictions on the validation set. True positives of 19,970 and true negatives of 14,824 were achieved by the model with only 892 false positives and 688 false negatives. From the distribution it is clear the model has a slight tendency of predicting correctly aligning with the positive class rate of 56.79% in the dataset. And the balanced distribution of the error (892 false positives against 688 false negatives) suggests that the model does not have a large bias toward either prediction class Table 5 Overall Confusion Matrix Actual Incorrect Predicted Incorrect Predicted Correct 14,824 (TN) 892 (FP) Actual Correct 688 (FN) 19,970 (TP) 4.4 Learned Modality Importance The adaptive gating mechanism which makes possible the learning of modality importance weights is one of the most significant contributions of the framework. When analysed across all validation samples the learned gate weights reveals that the affect modality had the highest average weight of 43.67%, indicating that when available the affective state information is the most predictive feature for knowledge tracing. The second highest weight of 26.32%, was recorded by the behavioural modality confirming the importance of help seeking patterns and persistent behaviours in predicting student performance Table 6 Learned Modality Importance Weights Modality Mean Weight Std Dev Percentage Affect 0.4367 ± 0.0668 43.7% Behavioral 0.2632 ± 0.0088 26.3% Text 0.1539 ± 0.0231 15.4% Math 0.0759 ± 0.0354 7.6% Code 0.0704 ± 0.0235 7.0% The text modality representing hint and scaffold usage patterns had an average weight of 15.39% showing that help seeking using textual resources gives meaningful signals when predicting student success. The math modality had weight of 7.59% suggesting that in cases where there is richer multimodal data, simple correctness knowledge estimates provide less marginal value. The code modality had the lowest weight of 7.04%. This is because CSEDM is the only dataset with code features and the model learns to down weigh it in sequences where code is not available 4.5 Training Dynamics and Convergence A stable convergence of the training process was achieved at 30 epochs with consistent improvement in the training and validation metrics. A decrease in the training loss was observed from 0.5012 in the first epoch to 0.0937 at the final epoch, representing a reduction of 81.3%. In a similar way the validation loss decreased from 0.2946 to 0.0957 indicating that the model has good generalization and does not show significant overfitting. There was equally an improvement in the validation AUC from 0.9444 in the first epoch to 0.9944 by epoch 29 and the best model checkpoint was saved at this epoch. 4.6 Cross-Validation Results To make sure that the results of the experiment robust and reproducible, a 5-fold stratified cross-validation was conducted on the complete dataset of 1,065 sequences. Each of the folds had about 213 validation sequences and the remaining 852 sequences were used for training. The results from cross-validation are consistent for all folds with AUC values ranging from 0.9923 to 0.9940 and a mean AUC of 0.9933 with a standard deviation of only 0.0006. These performance metrics are reliable and not as a result of data splits as is evidenced from the low coefficient of variation at 0.06% Table 7 5-Fold Cross-Validation Results Fold AUC Accuracy F1 Score RMSE Val Size 1 0.9931 95.38% 0.9596 0.1801 213 2 0.9934 95.54% 0.9612 0.1781 213 3 0.9937 95.56% 0.9620 0.1757 213 4 0.9923 95.09% 0.9570 0.1860 213 5 0.9940 95.76% 0.9633 0.1745 213 Mean ± Std 0.9933 ± 0.0006 95.47 ± 0.22% 0.9606 ± 0.0022 0.1789 ± 0.0041 1,065 total 5 CONCLUSION This study successfully designed and evaluated the Context-Aware Hierarchical Fusion with Learned Gating (CAHF-LG) framework, a Multimodal Deep Knowledge Tracing architecture with the aim to enable dynamic personalization of STEAM learning experiences. At the core of the framework are five distinct modalities of student interaction data including mathematics knowledge estimates, programming related features, text-based help seeking behaviour, affective states and behavioural features brought together through a three-layer hierarchical fusion strategy. Through this design the model has delivered strong predictive performance and also offers insights into the factors that influence student learning outcomes. The most notable quantitative outcome of the research is the high predictive accuracy achieved by CAHF-LG. When evaluated on a combined dataset with 1,065 student interaction sequences drawn from three structurally different platform datasets, the model reached an AUC-ROC of 0.9944. This result indicates a near perfect discrimination between correct and incorrect student responses. In comparison traditional Bayesian Knowledge Tracing usually reports AUC values between 0.65 and 0.75, Deep Knowledge Tracing ranges from about 0.80 to 0.86 while Self-Attentive Knowledge Tracing models generally fall between 0.82 and 0.88. On the other hand, Deep Knowledge Tracing ranges reports AUC values from about 0.80 to 0.86 and Self-Attentive Knowledge Tracing models between 0.82 and 0.88. How robust the CAHF-LG model is, is confirmed by a five-fold cross-validation, which produced a mean AUC of 0.9933 with a small standard deviation of 0.0006, highlighting the model’s stability across different data splits. Apart from the overall accuracy, framework provides insight into how different data modalities contribute to learning prediction. This is made possible because of the models learned gating mechanism which allows the model to infer relative importance for each modality. The results show that affective state features are more influential as they account for 43.7% of the total modality weight. Behavioural data was second in importance with 26.3% confirming the importance of persistence and help-seeking behaviours. On the other hand, traditional cognitive indicators like mathematical knowledge estimates and code progression had lower weights of 7.6% and 7.0% suggesting that when richer multimodal signals are available, correctness alone offers little insight. The framework showed strong generalization across the three datasets with AUC ranging between 0.8957 and 0.9986. When taken as a whole, these results show that a unified multimodal architecture can effectively adapt to different contexts, pedagogical settings, and data availability conditions, a very important consideration for resource constrained educational environments. Taken together, these results show that a unified multimodal architecture can adapt effectively to different subjects, pedagogical settings, and data availability conditions-an essential property for deployment in resource-constrained educational environments. Declarations Ethics Approval: Not applicable Conflict of Interest: The authors declare no conflict of interest Consent to participate: Not applicable. Consent to Publish: Not applicable. Funding This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Author Contribution Author contributions: WNM. conceptualised the study, conducted data analysis, developed the models, and wrote the manuscript. A.Z hosted the research at his institution, providing critical support and valuable insights throughout the study and reviewed the manuscript. Both authors reviewed and approved the final version of the manuscript. Acknowledgments The author gratefully acknowledges the guidance and support of Prof. A. Zimba throughout this research. Thanks are also extended to ZCAS University for providing the academic environment conducive to this work. The author appreciates the creators of the ASSISTments, CSEDM, and XES3G5M datasets for making these valuable resources publicly available for educational research. Data Availability The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request References Cunha MN. Active Learning Students - A State of the Art about STEM Education, in Current Research in Language, Literature and Education Vol. 6 , Book Publisher International (a part of SCIENCEDOMAIN International), 2022, pp. 94–99. 10.9734/bpi/crlle/v6/16079D Lytras M, Marouli C, Papadopoulou P. Best Practices in STEM Education: Using Active Learning and Novel Teaching Methodologies in Education for Innovation and Sustainability. Sandrone S, Scott G, Anderson WJ, Musunuru K. Active learning-based STEM education for in-person and online learning. Cell. Mar. 2021;184(6):1409–14. 10.1016/j.cell.2021.01.045 . Phiri A, Mwaanga C. Gender-based assessment of Science, Technology and Innovation ecosystem in Zambia. Oladele JI, Ndlovu M, Ayanwale MA. Computer Adaptive-Based Learning and Assessment for Enhancing STEM Education in Africa: A Fourth Industrial Revolution Possibility. In: Chirinda B, Luneta K, Uworwabayeho A, editors. in Mathematics Education in Africa. Cham: Springer International Publishing; 2022. pp. 131–44. 10.1007/978-3-031-13927-7_8 . Barrett AMB, Tikly LT, Joubert MJ, Gardner VG. Approaches to Strengthening Secondary STEM & ICT Education in Sub-Saharan Africa, Mastercard Foundation, Toronto, Ontario Canada, Mar. 2019. 10.15868/socialsector.35787 Cui Y, Chu M-W, Chen F. Analyzing Student Process Data in Game-Based Assessments with Bayesian Knowledge Tracing and Dynamic Bayesian Networks. J Educational Data Min. 2019;11(1):80–100. Lipton ZC, Berkowitz J, Elkan C. A Critical Review of Recurrent Neural Networks for Sequence Learning, Oct. 17, 2015, arXiv : arXiv:1506.00019. 10.48550/arXiv.1506.00019 Piech C et al. Deep Knowledge Tracing, 2015. Abdrakhmanov R, Zhaxanova A, Karatayeva M, Niyazova GZ, Berkimbayev K, Tuimebayev A. Development of a Framework for Predicting Students’ Academic Performance in STEM Education using Machine Learning Methods. IJACSA. 2024;15(1). 10.14569/IJACSA.2024.0150105 . Soloviev V. Machine learning approach for student engagement automatic recognition from facial expressions. Sci Pub Univ Novi Pazar Ser A. 2018;10(2):79–86. 10.5937/SPSUNP1802079S . Ciolacu M, Tehrani AF, Beer R, Popp H. Education 4.0 - Fostering Student’s Performance with Machine Learning Methods, 2017. Khan MA, Khojah M, Vivek. Artificial Intelligence and Big Data: The Advent of New Pedagogy in the Adaptive E-Learning System in the Higher Educational Institutions of Saudi Arabia, Education Research International , vol. 2022, pp. 1–10, Feb. 2022. 10.1155/2022/1263555 Khanal SS, Prasad PWC, Alsadoon A, Maag A. A systematic review: machine learning based recommendation systems for e-learning. Educ Inf Technol. July 2020;25(4):2635–64. 10.1007/s10639-019-10063-9 . Ong AKS. A Machine Learning Ensemble Approach for Predicting Factors Affecting STEM Students’ Future Intention to Enroll in Chemistry-Related Courses, Sustainability , vol. 14, no. 23, p. 16041, Dec. 2022, 10.3390/su142316041 Zafari M, Koochi F, Sadeghi-Niaraki A, Choi S-M, Tamer A. Implementation of a Machine Learning Approach to Model and Assess Student Spatial Intelligence for Advancing STEM Education. May. 2023;16. 10.21203/rs.3.rs-2918620/v1 . Blikstein P, Worsley M. Learn Analytics. Sept. 2016;3(2):220–38. 10.18608/jla.2016.32.11 . Multimodal Learning Analytics and Education Data Mining: using computational technologies to measure complex learning tasks. Chango W, Lara JA, Cerezo R, Romero C. A review on data fusion in multimodal learning analytics and educational data mining. WIREs Data Min Knowl. July 2022;12(4):e1458. 10.1002/widm.1458 . Ryoo J, Winkelmann K, editors. Innovative Learning Environments in STEM Higher Education: Opportunities, Challenges, and Looking Forward . in SpringerBriefs in Statistics. Cham: Springer International Publishing; 2021. 10.1007/978-3-030-58948-6 . Glancy AW, Moore TJ. Theoretical Foundations for Effective STEM Learning Environments, 2013. Valverde-Berrocoso J, Garrido-Arroyo MDC, Burgos-Videla C, Morales-Cevallos MB. Trends in Educational Research about e-Learning: A Systematic Literature Review (2009–2018). Sustainability. June 2020;12(12):5153. 10.3390/su12125153 . Pawłowski M, Wróblewska A, Sysko-Romańczuk S. Sensors. Feb. 2023;23(5):2381. 10.3390/s23052381 . Effective Techniques for Multimodal Data Fusion: A Comparative Analysis. Barnum G, Talukder S, Yue Y. On the Benefits of Early Fusion in Multimodal Representation Learning, Nov. 14, 2020, arXiv : arXiv:2011.07191. 10.48550/arXiv.2011.07191 Jiao T, Guo C, Feng X, Chen Y, Song J. A Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications. CMC. 2024;80(1):1–35. 10.32604/cmc.2024.053204 . Song X, et al. Cross-modal attention for multi-modal image registration. Med Image Anal. Nov. 2022;82:102612. 10.1016/j.media.2022.102612 . Liu T, Hu Y, Gao J, Sun Y, Yin B. Hierarchical Multi-Modal Transformer for Cross-Modal Long Document Classification. IEEE Trans Multimedia. 2025;27:8981–94. 10.1109/TMM.2025.3608295 . Jin Y, Peng J, Lin X, Yuan H, Wang L, Zheng C. Multimodal transformers are hierarchical modal-wise heterogeneous graphs, in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2025, pp. 2188–2209. Accessed: Feb. 21, 2026. [Online]. Available: https://aclanthology.org/2025.acl-long.109/ Baltrusaitis T, Ahuja C, Morency L-P. Multimodal Machine Learning: A Survey and Taxonomy, IEEE Trans. Pattern Anal. Mach. Intell. , vol. 41, no. 2, pp. 423–443, Feb. 2019, 10.1109/TPAMI.2018.2798607 Yang C, Liang Z, Yan D, Hu Z, Wu T. Hgtfm: Hierarchical gating-driven transformer fusion model for robust multimodal sentiment analysis, IEEE Access , 2025, Accessed: Feb. 21, 2026. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10965686/ Jiao T, Guo C, Feng X, Chen Y, Song J. A Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications. CMC. 2024;80(1):1–35. 10.32604/cmc.2024.053204 . Ding X, Han T, Fang Y, Larson E. An Approach for Combining Multimodal Fusion and Neural Architecture Search Applied to Knowledge Tracing, Nov. 08, 2021, arXiv : arXiv:2111.04497. 10.48550/arXiv.2111.04497 Liu D, Zhang Y, Zhang J, Li Q, Zhang C, Yin Y. Multiple features fusion attention mechanism enhanced deep knowledge tracing for student performance prediction. IEEE Access. 2020;8:194894–903. 10.1109/ACCESS.2020.3033200 . Praveen Kumar B, Kalpana AV, Nalini S. Gated Attention Based Deep Learning Model for Analyzing the Influence of Social Media on Education, Journal of Experimental & Theoretical Artificial Intelligence , vol. 37, no. 2, pp. 291–305, Feb. 2025, 10.1080/0952813X.2023.2188262 Zhang R, Chi X, Zhang W, Liu G, Wang D, Wang F. Unimodal training-multimodal prediction: Cross-modal federated learning with hierarchical aggregation, IEEE Transactions on Mobile Computing , 2025, Accessed: Feb. 21, 2026. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10989558/ Mostafaei SH, Tanha J, Sharafkhaneh A. A novel deep learning model based on transformer and cross modality attention for classification of sleep stages. J Biomed Inform. 2024;157:104689. Pandey S, Karypis G. A Self-Attentive model for Knowledge Tracing, July 16, 2019, arXiv : arXiv:1907.06837. 10.48550/arXiv.1907.06837 Abdelrahman G, Abdelfattah S, Wang Q, Lin Y. DBE-KT22: A Knowledge Tracing Dataset Based on Online Student Evaluation, Jan. 28, 2023, arXiv : arXiv:2208.12651. 10.48550/arXiv.2208.12651 Liu Z et al. XES3G5M: A Knowledge Tracing Benchmark Dataset with Auxiliary Information. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Reviews received at journal 17 May, 2026 Reviewers agreed at journal 07 May, 2026 Reviewers invited by journal 06 May, 2026 Editor assigned by journal 03 Apr, 2026 Submission checks completed at journal 01 Apr, 2026 First submitted to journal 01 Apr, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9232899","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":636356572,"identity":"52412bf7-5576-4e10-9bad-5a6f1d757184","order_by":0,"name":"Wilson Ng'andu Mwiiya","email":"","orcid":"","institution":"ZCAS University","correspondingAuthor":false,"prefix":"","firstName":"Wilson","middleName":"Ng'andu","lastName":"Mwiiya","suffix":""},{"id":636356573,"identity":"87281bb0-cc3d-4341-9611-0664f8d467be","order_by":1,"name":"Aaron Zimba","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA9UlEQVRIiWNgGAWjYBADHgYJBsYHMJ4EsVqYDUjSAlLGJoFg4wH87WcfPi5gqJPRnd38rPJHjZ29wQHmg7d5GOqicZp9Jt3YeAYDG4/ZnWNmNySOJSduOMCWbM3DwJbbgEvPgTQ2aR4GHh6zGwlmNwwbmBMMDvCYgURwapE//wykRQKoJf1bQWJDPdBh/N9AIji1GNwA22IA1JJjxnCw4TDjhgM8YBGcWgxvPGM25jFIAPrlTLFkw7HjiTMPsxlbzjFIwKlF7nwa42Oeijp7s9vtGz/+qKm25zve/PDGm4o63N6HOA+Zw4whMgpGwSgYBaOAVAAA/FlMH3lWvCIAAAAASUVORK5CYII=","orcid":"","institution":"ZCAS University","correspondingAuthor":true,"prefix":"","firstName":"Aaron","middleName":"","lastName":"Zimba","suffix":""}],"badges":[],"createdAt":"2026-03-26 10:38:17","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9232899/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9232899/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":109328893,"identity":"1f090c13-e1cd-42b8-a62c-09953a8fa41c","added_by":"auto","created_at":"2026-05-15 15:32:50","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":118674,"visible":true,"origin":"","legend":"\u003cp\u003eTraining and Validation Loss/AUC Curves over 30 Epochs\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-9232899/v1/92acd22d37bbdad3a852619a.png"}],"financialInterests":"No competing interests reported.","formattedTitle":"A Multimodal Approach to Knowledge Tracing for Personalized Learning based on Cognitive and Affective Data in Resource-Constrained Environments","fulltext":[{"header":"1 INTRODUCTION","content":"\u003ch2\u003e1.1\u0026nbsp; \u0026nbsp; \u0026nbsp;Background and Educational Context\u003c/h2\u003e\n\u003cp\u003ePersonalized learning has become a central objective in higher education, particularly in Science, Technology, Engineering, Arts, and Mathematics (STEAM) disciplines where learners exhibit diverse cognitive abilities, learning behaviours, and emotional responses. STEAM education plays a critical role in developing problem-solving skills and innovation capacity required for addressing real-world challenges [1] - [3]. However, in resource-constrained environments such as Zambia, educational systems face persistent challenges including limited teaching capacity, high learner-to-instructor ratios, and inadequate infrastructure [4]. These constraints often result in the adoption of one-size-fits-all instructional approaches that do not adequately account for individual learner differences, including variations in learning styles and affective states [5], [6]. Consequently, many students experience disengagement, poor academic performance, and reduced retention in STEAM programmes, highlighting the need for scalable and adaptive learning solutions.\u003c/p\u003e\n\u003cp\u003eKnowledge tracing (KT) has emerged as a key technique in learning analytics for modelling students\u0026rsquo; evolving knowledge states based on interaction data. Traditional approaches such as Bayesian Knowledge Tracing (BKT) [7] and Deep Knowledge Tracing (DKT) [9] have demonstrated success in predicting student performance; however, they predominantly rely on correctness-based cognitive signals and overlook the broader multimodal nature of the learning process. Educational research shows that learning outcomes are influenced not only by cognitive factors but also by affective states and behavioural patterns [19] - [21], while multimodal learning analytics emphasizes the integration of diverse data sources to capture complex learning dynamics [17], [18]. Despite these advances, existing multimodal approaches often employ static or simplistic fusion strategies that fail to capture the dynamic and context-dependent relationships among modalities. To address this gap, this study proposes a multimodal approach to knowledge tracing that integrates cognitive and affective data through a context-aware deep learning framework, enabling more accurate and adaptive modelling of learner behaviour in resource-constrained educational environments.\u003c/p\u003e\n\u003ch2\u003e1.2\u0026nbsp; \u0026nbsp; \u0026nbsp;Knowledge Tracing and Personalized Learning\u003c/h2\u003e\n\u003cp\u003eKnowledge tracing represents a data driven approach to modelling the evolving knowledge states of students based on interaction sequences. Traditional KT approaches like Bayesian Knowledge Tracing (BKT), inspired by the concept of mastery learning, model student knowledge as a \u0026nbsp; binary latent variable [7] hidden from direct observation and usually achieve an AUC-ROC score of between 0.65 and 0.75 on benchmark datasets\u003c/p\u003e\n\u003cp\u003eThe limitations of foundational BKT models, success in deep learning [8] and the need to model more complex patterns inspired the development of Deep Knowledge Tracing (DKT) [9]. DKT makes use of recurrent neural networks to capture complex temporal dependencies in learning sequencies and has an improved AUC-ROC score ranging from 0.80-0.86.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eDespite these advances in data driven solutions [10], [11], [12], [13], [14], [15], [16], the notable foundational Knowledge Tracing models rely on simple correctness sequences and overlook the rich multimodal nature of the learning process. According to [17], the complex process of learning, especially in the STEAM domain involves problem solving, collaboration, experimentation which produce information that cannot simply be captured by correctness alone [18]. Educational psychology research has it that learning outcomes are not only influenced by cognitive factors but also affective states of learners [19], [20], [21]. This limitation led to the emergence of multimodal learning analytics, which argues that the fusion of diverse data streams including behavioural patterns, textual responses, domain-specific content, and emotional states present a holistic view of the learning process.\u003c/p\u003e\n\u003cp\u003eThis study makes the following contributions:\u003c/p\u003e\n\u003cp\u003e1.\u0026nbsp; \u0026nbsp;A multimodal knowledge tracing framework that integrates cognitive, behavioural, and affective data to model learner knowledge states more comprehensively than traditional cognition-focused approaches.\u003c/p\u003e\n\u003cp\u003e2.\u0026nbsp; \u0026nbsp;A context-aware hierarchical fusion mechanism with adaptive gating that dynamically learns the relative importance of multiple modalities, enabling robust performance even when some data sources are missing-an important requirement for resource-constrained environments.\u003c/p\u003e\n\u003cp\u003e3.\u0026nbsp; \u0026nbsp;Empirical evidence on the role of affective and behavioural factors in learning, demonstrating that non-cognitive signals contribute more significantly to performance prediction than traditional correctness-based indicators.\u003c/p\u003e\n\u003cp\u003e4.\u0026nbsp; \u0026nbsp;A scalable and generalisable approach validated across multiple educational datasets, showing strong predictive performance and applicability across different learning domains, including mathematics and programming.\u003c/p\u003e\n\u003cp\u003eThe remainder of this paper is structured as follows. Section 2 reviews related work on multimodal learning analytics and knowledge tracing. Section 3 presents the proposed methodology and model architecture. Section 4 reports the experimental results and analysis. Section 5 discusses the findings and implications, and Section 6 concludes the paper with directions for future research.\u003c/p\u003e"},{"header":"2 RELATED WORK","content":"\u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Multimodal Fusion Mechanisms in Learning Systems\u003c/h2\u003e \u003cp\u003eThe fusion of multimodal data in deep learning models has become a central problem in artificial intelligence research. Traditional approaches to multimodal fusion techniques in education fall in three broad categories based on the point in the pipeline where integration happens: early (feature level), late (decision level) and intermediate (hybrid) fusion [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. Early fusion approaches [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e], [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e] assume homogeneous feature representations, which is rarely valid in educational data characterized by heterogeneous modalities such as affective states and behavioural signals. Late fusion methods [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e], [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e] preserve modality-specific information but fail to capture cross-modal dependencies critical for modelling learning processes. Intermediate fusion strategies, including attention-based models [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e], provide improved integration but often lack hierarchical structure and contextual adaptation. For example,[\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e] proposed a hierarchical transformer used for classification of documents, however, the fusion strategy that was used was not optimized for temporal educational data. And in a similar manner, [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e] models modalities as heterogeneous graphs, however, the study does not incorporate contextual gating for adaptive fusion. According to [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e], intermediate fusion outperforms early and late fusion in a number of learning scenarios as they are able to capture additional cross modal information while maintaining their individual characteristics. The field has however, undergone a notable shift by focusing on more implicit approaches that modulate importance of each modality based on context and demand. A number of recent research studies have refined the intermediate fusion into a number of categories reflecting modern architectural choices [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e], [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e], [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e], [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]. The single level fusion combines modalities at a specific network layer and the hierarchical fusion integrates modalities across multiple depths in a network while the attention mechanism assigns dynamic learned weights to different modalities. However, these approaches rarely integrate multimodal signals into knowledge tracing frameworks.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Gating Mechanisms and Context-Aware Modelling\u003c/h2\u003e \u003cp\u003eThe gated fusion mechanism, a family of neural network architectures that makes use of multiplicative gates to control how information from multiple sources or modalities is combined has been used to dynamically weigh modalities [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e]. The adaptive weighting makes it possible for the model to prioritize informative modalities. An example of this idea was proposed by [\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e] as the Gated Multimodal Unit (GMU), laying foundation for gated multimodal fusion. Mostly, these techniques rely on modality specific features and fail to condition the weights on external contextual signals like time elapsed or difficult level. A hierarchical aggregation for federated learning in educational data is introduced by [\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e] but does not address context adaptation. Other studies like [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e], [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e] have made use of cross modal attention, however, the gating strategy used is neither hierarchical nor sensitive to context. Context aware fusion builds on adaptive gating [\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e] by allowing the fusion process to respond to contextual information and the Context-Aware Attentive Knowledge Tracing (AKT) model by [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e] highlights the benefits of including context in educational knowledge tracing. As a result the model showed a 6% improvement in AUC on standard benchmark datasets and maintained interpretability through clearly defined attention weights. Existing approaches do not simultaneously model hierarchical fusion, context-awareness, and multimodal knowledge tracing, particularly in resource-constrained educational environments.\u003c/p\u003e \u003cp\u003eDespite significant advances in multimodal learning analytics, existing approaches exhibit three key limitations. First, most knowledge tracing models remain predominantly cognitive, neglecting the role of affective and behavioural factors in learning. Second, current multimodal fusion techniques often rely on static or shallow integration strategies that fail to capture complex and hierarchical relationships among modalities. Third, limited attention has been given to context-aware adaptive mechanisms that can dynamically adjust modality importance, particularly in resource-constrained educational settings where data availability may be inconsistent. These limitations highlight the need for a unified framework that integrates multimodal data through context-aware and hierarchical fusion to support more accurate and scalable personalized learning.\u003c/p\u003e \u003c/div\u003e"},{"header":"3 METHODS","content":"\u003cp\u003eThree benchmark datasets were used in the study. These datasets include ASSISTments2015, CSEDM19 and XES3G5M. The ASSISTments2015 dataset retrieved from ASSISTments skill builder problem sets, has detailed records of student interactions with an intelligent tutoring system, which includes student responses to class assessment activities, problem solving attempts, and performance data. The dataset contains roughly 708,631 interactions from over 19,917 students and 100 skills [\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e] providing a robust foundation for training and evaluating DKT models.\u003c/p\u003e \u003cp\u003eA code specific datasets CSEDM19 was used to test the ability of the framework in handling programming data. This dataset from the CSEDM Challenge provided code submissions from programming tasks and is ideal for knowledge tracing. One mathematics specific datasets XES3G5M was used to demonstrate generalizability of the model beyond ASSISTments. The XES3G5M [\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e] dataset is a large public benchmark dataset that contains 5.5\u0026nbsp;million student interactions with 7,652 math questions and 865 knowledge components. The dataset has rich contextual information which includes text question, solution analysis and a tree structured knowledge component relation. Two types of question are used, multiple choice (1,510 questions) and fill in the blank (6,142 questions)\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Context-Aware Hierarchical Fusion with Learned Gating Architecture\u003c/h2\u003e \u003cdiv id=\"Sec8\" class=\"Section3\"\u003e \u003ch2\u003e3.1.1 Modality Encoding and Representation Learning\u003c/h2\u003e \u003cp\u003eThe CAHF-LG model takes in five distinct modalities as input, each modality captures different aspects of the student learning behaviour. These modalities are processed using dedicated encoder networks projecting these heterogeneous inputs to a unified 128-dimensional latent space enabling cross modal attention and fusion operations. Each of the modality encoders is made up of a two-layer neural network that uses layer normalization with GELU activation functions. To represent the encoding modality mathematically,\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e\n$$\\:{h}_{m}=\\text{LayerNorm}\\left(\\text{GELU}\\left(\\text{LayerNorm}\\left({W}_{2}\\cdot\\:\\text{GELU}\\left({W}_{1}\\cdot\\:{x}_{m}+{b}_{1}\\right)+{b}_{2}\\right)\\right)\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\right(1)$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere W₁ has dimensions 128 \u0026times; dₘ, W₂ is 128 \u0026times; 128, and xₘ is just the raw input.\u003c/p\u003e \u003cp\u003eThis setup projects each modality into a common representational space making it possible for comparisons and interactions cross modalities and preserves modality unique characteristics through learned non linear transformations\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eInput Modalities and Dimensions\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModality\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDimension\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eDescription\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMath\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eKnowledge estimate from BKT (Ln) or running accuracy\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCode\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProgramming event type score (Submit, Compile, Edit)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eText\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHint and scaffold usage patterns\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAffect\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAffective states: Bored, Concentrating, Confused, Frustrated, Off-task, Gaming\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBehavioral\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHelp-seeking, attempt persistence, time management\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section3\"\u003e \u003ch2\u003e3.1.2 Context Encoding Mechanism\u003c/h2\u003e \u003cp\u003eThe context encoder generates a detailed picture of where the learner stands and this picture, through the gating mechanisms dynamically adjusts how information flows. The Context Encoder does not treat the learner\u0026rsquo;s context as static metadata but dynamically constructs rich context representation that includes the context features and the current affective state of the learner. This approach creates a rich context vector that is then used to condition the gating mechanisms at all the three fusion levels. This in effect creates an adaptive multimodal processing.\u003c/p\u003e \u003cp\u003eThe base context vector is made of four components, 1) problem type encoding (0.0 for math, 0.5 for mixed, 1.0 for code), 2) student proficiency estimated as the running average of correctness, 3) learning phase indicating position in the sequence normalized to [0,1], and 4) task difficulty estimated as the inverse of student performance. these four scalar values are the concatenated with the affect context embedding resulting from the computation of the mean affective state across the sequence. This operation produces a 128-dimensional context vector capturing both the stable learner characteristics and the dynamic emotional state\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section3\"\u003e \u003ch2\u003e3.1.3 Three-Level Hierarchical Fusion Architecture\u003c/h2\u003e \u003cp\u003eThe main architectural contribution of the CAHF-LG is in its three-level hierarchical fusion design. The multimodal information is processed at three distinct granularities by the hierarchical fusion mechanism. This gradually integrates features from the individual modalities to temporal sequences. Each level includes context-aware gating to make sure that the fusion adapts to the specific learner context at every stage of processing.\u003c/p\u003e \u003cp\u003eThe first fusion level is the feature level fusion which operates on each of the modalities independently and applies self-attention to capture intra model dependencies. For a given modality \u003cem\u003em\u003c/em\u003e, the feature matric that is encoded \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{h}_{m}\\in\\:{\\mathbb{ℝ}}^{T\\times\\:128}\\:\\)\u003c/span\u003e\u003c/span\u003eis first adjusted through a context driven gating mechanism where the gate:\u003cdiv id=\"Equb\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equb\" name=\"EquationSource\"\u003e\n$$\\:{g}_{m}=\\sigma\\:\\left({W}_{g}\\cdot\\:\\text{context}\\right)\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\:\\left(2\\right)$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere σ is the sigmoid activation that regulates feature importance.\u003c/p\u003e \u003cp\u003eThese gated features then go through a multi-head self-attention which is later on followed by a position-wise feedforward network. This fusion level helps the model to identify the most relevant features within each modality given the current learner context.\u003c/p\u003e \u003cp\u003eThe second level of fusion involves integrating information from all the five modalities. To be able to learn how important each modality is given a particular context, an adaptive gating mechanism was used. The concatenation of pooled modalities and the context vector are fed as input to the adaptive gate network which produces an output of five-dimensional weight vector:\n\u003cp\u003e\u003cimg src=\"https://myfiles.space/user_files/127393_c7e80a1c9bb65875/127393_custom_files/img1778858959.png\" style=\"width: 535px;\"\u003e\u003c/p\u003e \u003cp\u003ewith \u003cem\u003ef\u003c/em\u003e as the three layer multi layer perceptron MLP,\u003c/p\u003e \u003cp\u003ebase_weight being the learned prior weights,\u003c/p\u003e \u003cp\u003eand τ is a trainable temperature parameter.\u003c/p\u003e \u003cp\u003eThese weights are used to compute the weighted sum of the modality representations, resulting into a fused multimodal embedding. In addition, before this, aggregation, a cross-modal attention with four heads is applied, as it allows modalities to exchange information before the final weighted combination.\u003c/p\u003e \u003cp\u003eThe third fusion level is the temporal level fusion which models the temporal dependencies across the learning sequencies by making use of a Transformer encoder with causal masking. This fusion level has a temporal fusion module that is made of four transformer layers with each layer having 8 attention heads and processing sequencies of up to 200 timesteps. To make sure that predictions at a particular timestep t only attends to information from timesteps 1 to t-1, causal masking is used. This approach maintains the proper autoregressive structure needed for knowledge tracing. Then finally Learnable positional encodings are added to capture sequential ordering information\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section3\"\u003e \u003ch2\u003e3.1.4 Adaptive Gating Mechanism\u003c/h2\u003e \u003cp\u003eThe adaptive gating mechanism is the main component that enables context-aware multimodal fusion. Instead of relying on a fixed attention mechanism, the adaptive gate module learns to produce dynamic weights that would adapt based on both input features and leaner context. The gating network architecture is made of three connected layers with GELU activations, gradually reducing dimensionality of input until it matches that of the items being gated Linear(context_dim\u0026thinsp;+\u0026thinsp;hidden_dim \u0026rarr; hidden_dim) \u0026rarr; GELU \u0026rarr; Dropout \u0026rarr; Linear(hidden_dim \u0026rarr; hidden_dim/2) \u0026rarr; GELU \u0026rarr; Linear(hidden_dim/2 \u0026rarr; num_items)\u003c/p\u003e \u003cp\u003eThe output is merged with learnable base weights encoding prior assumptions about the relevance of modalities and then adjusting them using a learnable temperature parameter before applying SoftMax normalization. These learnable base weights act as a form of regularisation, pushing the gating mechanism towards the preferred modality configurations when the input signal is not strong. During training, the learnable base weights are updated to capture the average contribution from each modality across the datasets. Low values encourage the model to focus on the few modalities and large values lead to a more uniform distribution. Because the temperature is optimized with the rest of the model parameters, it makes it possible for the system to automatically find an appropriate level of balance between specialization and flexibility for a task given\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThree-Level Hierarchical Fusion Specifications\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFusion Level\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMechanism\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePurpose\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLevel 1: Feature\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eContext-gated self-attention (4 heads)\u0026thinsp;+\u0026thinsp;FFN per modality\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCapture intra-modal dependencies; context-aware feature importance\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLevel 2: Modality\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAdaptive gating\u0026thinsp;+\u0026thinsp;cross-modal attention (4 heads)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLearn modality importance weights; enable cross-modal information flow\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLevel 3: Temporal\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTransformer encoder (4 layers, 8 heads) with causal mask\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eModel temporal evolution of knowledge state across learning sequence\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"4 RESULTS ANALYSIS","content":"\u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Overall Model Performance\u003c/h2\u003e \u003cp\u003eThe framework achieved exceptional predictive performance across all evaluation metrics. On the validation set, the model achieved an AUC-ROC of 0.9944 demonstrating its exceptional predictive performance. This result exceeds the performance of traditional knowledge tracing approaches by showing a near perfect discrimination between correct and incorrect student responses. The accuracy of the model was at 95.66% meaning the model could correctly predict student response outcome 95 out of every 100 interactions. An F1 score of 0.9619 was recorded confirming that the accuracy was balanced across the correct and incorrect response classes while the model maintained strong precision and recall. Additional insight into the quality of the predictions was obtained from the Root Mean Square Error (RMSE) of 0.1732. Such a low RMSE means that if the model predicts a correctness of 70% probability, then the actual observed correctness rate is close to 70%. Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e summarizes the primary performance metrics achieved by the MDKT CAHF-LG framework.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCAHF-LG Overall Performance Metrics\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMetric\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eValue\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAUC-ROC\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.9944\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e95.66%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eF1 Score\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.9619\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRMSE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.1732\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePrecision (Correct Class)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.96\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRecall (Correct Class)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.97\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Per-Dataset Performance\u003c/h2\u003e \u003cp\u003eThe performance of the CAHF-LG framework was also evaluated separately across the three structurally different datasets, each with distinct context and characteristics. This per dataset evaluation reveals how the unified multimodal architecture adapts to different educational domains and data availability patterns.\u003c/p\u003e \u003cp\u003eThe highest performance was achieved by the XES3G5M dataset with an AUC of 0.9986 and accuracy of 98.10%. This reflects the availability of large volume of training data and a consistent response pattern. The ASSISTments dataset had an AUC of 0.9878 and accuracy of 93.74% showing strong performance on the dataset with the richest features including affect state. Comparatively lower AUC performance of 0.8957 and accuracy of 92.31% was recorded from the CSEDM dataset. The low performance on this dataset is due to the small sample size with on 68 students and the variable nature of problem solving in programming tasks compared to mathematics\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePer-Dataset Performance Comparison\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDataset\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAUC-ROC\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePR-AUC\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSamples\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eASSISTments\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.9878\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e93.74%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.9812\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e19,707\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCSEDM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.8957\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e92.31%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.7734\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e533\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eXES3G5M\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.9986\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e98.10%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.9997\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e16,134\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e4.3 Confusion Matrix Analysis\u003c/h2\u003e \u003cp\u003eThe model correctly classified 34,794 instances (95.66% accuracy) out of a total of 36,374 predictions on the validation set. True positives of 19,970 and true negatives of 14,824 were achieved by the model with only 892 false positives and 688 false negatives. From the distribution it is clear the model has a slight tendency of predicting correctly aligning with the positive class rate of 56.79% in the dataset. And the balanced distribution of the error (892 false positives against 688 false negatives) suggests that the model does not have a large bias toward either prediction class\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eOverall Confusion Matrix\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003eActual Incorrect\u003c/b\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePredicted Incorrect\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePredicted Correct\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003e14,824 (TN)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e892 (FP)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eActual Correct\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e688 (FN)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e19,970 (TP)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003e4.4 Learned Modality Importance\u003c/h2\u003e \u003cp\u003eThe adaptive gating mechanism which makes possible the learning of modality importance weights is one of the most significant contributions of the framework. When analysed across all validation samples the learned gate weights reveals that the affect modality had the highest average weight of 43.67%, indicating that when available the affective state information is the most predictive feature for knowledge tracing. The second highest weight of 26.32%, was recorded by the behavioural modality confirming the importance of help seeking patterns and persistent behaviours in predicting student performance\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab6\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 6\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eLearned Modality Importance Weights\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModality\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMean Weight\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eStd Dev\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePercentage\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAffect\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.4367\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u0026plusmn;\u0026thinsp;0.0668\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e43.7%\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eBehavioral\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.2632\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u0026plusmn;\u0026thinsp;0.0088\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e26.3%\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eText\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.1539\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u0026plusmn;\u0026thinsp;0.0231\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e15.4%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMath\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.0759\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u0026plusmn;\u0026thinsp;0.0354\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.6%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCode\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.0704\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u0026plusmn;\u0026thinsp;0.0235\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.0%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe text modality representing hint and scaffold usage patterns had an average weight of 15.39% showing that help seeking using textual resources gives meaningful signals when predicting student success. The math modality had weight of 7.59% suggesting that in cases where there is richer multimodal data, simple correctness knowledge estimates provide less marginal value. The code modality had the lowest weight of 7.04%. This is because CSEDM is the only dataset with code features and the model learns to down weigh it in sequences where code is not available\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003e4.5 Training Dynamics and Convergence\u003c/h2\u003e \u003cp\u003eA stable convergence of the training process was achieved at 30 epochs with consistent improvement in the training and validation metrics. A decrease in the training loss was observed from 0.5012 in the first epoch to 0.0937 at the final epoch, representing a reduction of 81.3%. In a similar way the validation loss decreased from 0.2946 to 0.0957 indicating that the model has good generalization and does not show significant overfitting. There was equally an improvement in the validation AUC from 0.9444 in the first epoch to 0.9944 by epoch 29 and the best model checkpoint was saved at this epoch.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003e4.6 Cross-Validation Results\u003c/h2\u003e \u003cp\u003eTo make sure that the results of the experiment robust and reproducible, a 5-fold stratified cross-validation was conducted on the complete dataset of 1,065 sequences. Each of the folds had about 213 validation sequences and the remaining 852 sequences were used for training. The results from cross-validation are consistent for all folds with AUC values ranging from 0.9923 to 0.9940 and a mean AUC of 0.9933 with a standard deviation of only 0.0006. These performance metrics are reliable and not as a result of data splits as is evidenced from the low coefficient of variation at 0.06%\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab7\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 7\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e5-Fold Cross-Validation Results\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFold\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAUC\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eF1 Score\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eRMSE\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eVal Size\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.9931\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e95.38%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.9596\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.1801\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e213\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.9934\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e95.54%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.9612\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.1781\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e213\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.9937\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e95.56%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.9620\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.1757\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e213\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.9923\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e95.09%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.9570\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.1860\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e213\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.9940\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e95.76%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.9633\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.1745\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e213\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMean\u0026thinsp;\u0026plusmn;\u0026thinsp;Std\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.9933\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0006\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e95.47\u0026thinsp;\u0026plusmn;\u0026thinsp;0.22%\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.9606\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0022\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e0.1789\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0041\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e1,065 total\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"5 CONCLUSION","content":"\u003cp\u003eThis study successfully designed and evaluated the Context-Aware Hierarchical Fusion with Learned Gating (CAHF-LG) framework, a Multimodal Deep Knowledge Tracing architecture with the aim to enable dynamic personalization of STEAM learning experiences. At the core of the framework are five distinct modalities of student interaction data including mathematics knowledge estimates, programming related features, text-based help seeking behaviour, affective states and behavioural features brought together through a three-layer hierarchical fusion strategy. Through this design the model has delivered strong predictive performance and also offers insights into the factors that influence student learning outcomes.\u003c/p\u003e \u003cp\u003eThe most notable quantitative outcome of the research is the high predictive accuracy achieved by CAHF-LG. When evaluated on a combined dataset with 1,065 student interaction sequences drawn from three structurally different platform datasets, the model reached an AUC-ROC of 0.9944. This result indicates a near perfect discrimination between correct and incorrect student responses. In comparison traditional Bayesian Knowledge Tracing usually reports AUC values between 0.65 and 0.75, Deep Knowledge Tracing ranges from about 0.80 to 0.86 while Self-Attentive Knowledge Tracing models generally fall between 0.82 and 0.88. On the other hand, Deep Knowledge Tracing ranges reports AUC values from about 0.80 to 0.86 and Self-Attentive Knowledge Tracing models between 0.82 and 0.88. How robust the CAHF-LG model is, is confirmed by a five-fold cross-validation, which produced a mean AUC of 0.9933 with a small standard deviation of 0.0006, highlighting the model\u0026rsquo;s stability across different data splits.\u003c/p\u003e \u003cp\u003eApart from the overall accuracy, framework provides insight into how different data modalities contribute to learning prediction. This is made possible because of the models learned gating mechanism which allows the model to infer relative importance for each modality. The results show that affective state features are more influential as they account for 43.7% of the total modality weight. Behavioural data was second in importance with 26.3% confirming the importance of persistence and help-seeking behaviours. On the other hand, traditional cognitive indicators like mathematical knowledge estimates and code progression had lower weights of 7.6% and 7.0% suggesting that when richer multimodal signals are available, correctness alone offers little insight.\u003c/p\u003e \u003cp\u003eThe framework showed strong generalization across the three datasets with AUC ranging between 0.8957 and 0.9986. When taken as a whole, these results show that a unified multimodal architecture can effectively adapt to different contexts, pedagogical settings, and data availability conditions, a very important consideration for resource constrained educational environments. Taken together, these results show that a unified multimodal architecture can adapt effectively to different subjects, pedagogical settings, and data availability conditions-an essential property for deployment in resource-constrained educational environments.\u003c/p\u003e"},{"header":"Declarations","content":" \u003cp\u003e \u003cstrong\u003eEthics Approval:\u003c/strong\u003e \u003cp\u003eNot applicable\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eConflict of Interest:\u003c/strong\u003e \u003cp\u003eThe authors declare no conflict of interest\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eConsent to participate:\u003c/strong\u003e \u003cp\u003eNot applicable.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eConsent to Publish:\u003c/strong\u003e \u003cp\u003eNot applicable.\u003c/p\u003e \u003c/p\u003e\u003ch2\u003eFunding\u003c/h2\u003e \u003cp\u003eThis research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eAuthor contributions: WNM. conceptualised the study, conducted data analysis, developed the models, and wrote the manuscript. A.Z hosted the research at his institution, providing critical support and valuable insights throughout the study and reviewed the manuscript. Both authors reviewed and approved the final version of the manuscript.\u003c/p\u003e\u003ch2\u003eAcknowledgments\u003c/h2\u003e \u003cp\u003eThe author gratefully acknowledges the guidance and support of Prof. A. Zimba throughout this research. Thanks are also extended to ZCAS University for providing the academic environment conducive to this work. The author appreciates the creators of the ASSISTments, CSEDM, and XES3G5M datasets for making these valuable resources publicly available for educational research.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eThe datasets used and/or analysed during the current study are available from the corresponding author on reasonable request\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eCunha MN. Active Learning Students - A State of the Art about STEM Education, in \u003cem\u003eCurrent Research in Language, Literature and Education Vol. 6\u003c/em\u003e, Book Publisher International (a part of SCIENCEDOMAIN International), 2022, pp. 94\u0026ndash;99. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.9734/bpi/crlle/v6/16079D\u003c/span\u003e\u003cspan address=\"10.9734/bpi/crlle/v6/16079D\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLytras M, Marouli C, Papadopoulou P. Best Practices in STEM Education: Using Active Learning and Novel Teaching Methodologies in Education for Innovation and Sustainability.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSandrone S, Scott G, Anderson WJ, Musunuru K. Active learning-based STEM education for in-person and online learning. Cell. Mar. 2021;184(6):1409\u0026ndash;14. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.cell.2021.01.045\u003c/span\u003e\u003cspan address=\"10.1016/j.cell.2021.01.045\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePhiri A, Mwaanga C. Gender-based assessment of Science, Technology and Innovation ecosystem in Zambia.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOladele JI, Ndlovu M, Ayanwale MA. Computer Adaptive-Based Learning and Assessment for Enhancing STEM Education in Africa: A Fourth Industrial Revolution Possibility. In: Chirinda B, Luneta K, Uworwabayeho A, editors. in Mathematics Education in Africa. Cham: Springer International Publishing; 2022. pp. 131\u0026ndash;44. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/978-3-031-13927-7_8\u003c/span\u003e\u003cspan address=\"10.1007/978-3-031-13927-7_8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBarrett AMB, Tikly LT, Joubert MJ, Gardner VG. Approaches to Strengthening Secondary STEM \u0026amp; ICT Education in Sub-Saharan Africa, Mastercard Foundation, Toronto, Ontario Canada, Mar. 2019. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.15868/socialsector.35787\u003c/span\u003e\u003cspan address=\"10.15868/socialsector.35787\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCui Y, Chu M-W, Chen F. Analyzing Student Process Data in Game-Based Assessments with Bayesian Knowledge Tracing and Dynamic Bayesian Networks. J Educational Data Min. 2019;11(1):80\u0026ndash;100.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLipton ZC, Berkowitz J, Elkan C. A Critical Review of Recurrent Neural Networks for Sequence Learning, Oct. 17, 2015, \u003cem\u003earXiv\u003c/em\u003e: arXiv:1506.00019. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.48550/arXiv.1506.00019\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.1506.00019\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePiech C et al. Deep Knowledge Tracing, 2015.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbdrakhmanov R, Zhaxanova A, Karatayeva M, Niyazova GZ, Berkimbayev K, Tuimebayev A. Development of a Framework for Predicting Students\u0026rsquo; Academic Performance in STEM Education using Machine Learning Methods. IJACSA. 2024;15(1). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.14569/IJACSA.2024.0150105\u003c/span\u003e\u003cspan address=\"10.14569/IJACSA.2024.0150105\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSoloviev V. Machine learning approach for student engagement automatic recognition from facial expressions. Sci Pub Univ Novi Pazar Ser A. 2018;10(2):79\u0026ndash;86. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.5937/SPSUNP1802079S\u003c/span\u003e\u003cspan address=\"10.5937/SPSUNP1802079S\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCiolacu M, Tehrani AF, Beer R, Popp H. Education 4.0 - Fostering Student\u0026rsquo;s Performance with Machine Learning Methods, 2017.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKhan MA, Khojah M, Vivek. Artificial Intelligence and Big Data: The Advent of New Pedagogy in the Adaptive E-Learning System in the Higher Educational Institutions of Saudi Arabia, \u003cem\u003eEducation Research International\u003c/em\u003e, vol. 2022, pp. 1\u0026ndash;10, Feb. 2022. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1155/2022/1263555\u003c/span\u003e\u003cspan address=\"10.1155/2022/1263555\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKhanal SS, Prasad PWC, Alsadoon A, Maag A. A systematic review: machine learning based recommendation systems for e-learning. Educ Inf Technol. July 2020;25(4):2635\u0026ndash;64. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s10639-019-10063-9\u003c/span\u003e\u003cspan address=\"10.1007/s10639-019-10063-9\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOng AKS. A Machine Learning Ensemble Approach for Predicting Factors Affecting STEM Students\u0026rsquo; Future Intention to Enroll in Chemistry-Related Courses, \u003cem\u003eSustainability\u003c/em\u003e, vol. 14, no. 23, p. 16041, Dec. 2022, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/su142316041\u003c/span\u003e\u003cspan address=\"10.3390/su142316041\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZafari M, Koochi F, Sadeghi-Niaraki A, Choi S-M, Tamer A. Implementation of a Machine Learning Approach to Model and Assess Student Spatial Intelligence for Advancing STEM Education. May. 2023;16. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.21203/rs.3.rs-2918620/v1\u003c/span\u003e\u003cspan address=\"10.21203/rs.3.rs-2918620/v1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBlikstein P, Worsley M. Learn Analytics. Sept. 2016;3(2):220\u0026ndash;38. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.18608/jla.2016.32.11\u003c/span\u003e\u003cspan address=\"10.18608/jla.2016.32.11\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Multimodal Learning Analytics and Education Data Mining: using computational technologies to measure complex learning tasks.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChango W, Lara JA, Cerezo R, Romero C. A review on data fusion in multimodal learning analytics and educational data mining. WIREs Data Min Knowl. July 2022;12(4):e1458. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1002/widm.1458\u003c/span\u003e\u003cspan address=\"10.1002/widm.1458\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRyoo J, Winkelmann K, editors. \u003cem\u003eInnovative Learning Environments in STEM Higher Education: Opportunities, Challenges, and Looking Forward\u003c/em\u003e. in SpringerBriefs in Statistics. Cham: Springer International Publishing; 2021. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/978-3-030-58948-6\u003c/span\u003e\u003cspan address=\"10.1007/978-3-030-58948-6\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGlancy AW, Moore TJ. Theoretical Foundations for Effective STEM Learning Environments, 2013.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eValverde-Berrocoso J, Garrido-Arroyo MDC, Burgos-Videla C, Morales-Cevallos MB. Trends in Educational Research about e-Learning: A Systematic Literature Review (2009\u0026ndash;2018). Sustainability. June 2020;12(12):5153. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/su12125153\u003c/span\u003e\u003cspan address=\"10.3390/su12125153\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePawłowski M, Wr\u0026oacute;blewska A, Sysko-Romańczuk S. Sensors. Feb. 2023;23(5):2381. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/s23052381\u003c/span\u003e\u003cspan address=\"10.3390/s23052381\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Effective Techniques for Multimodal Data Fusion: A Comparative Analysis.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBarnum G, Talukder S, Yue Y. On the Benefits of Early Fusion in Multimodal Representation Learning, Nov. 14, 2020, \u003cem\u003earXiv\u003c/em\u003e: arXiv:2011.07191. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.48550/arXiv.2011.07191\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2011.07191\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJiao T, Guo C, Feng X, Chen Y, Song J. A Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications. CMC. 2024;80(1):1\u0026ndash;35. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.32604/cmc.2024.053204\u003c/span\u003e\u003cspan address=\"10.32604/cmc.2024.053204\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSong X, et al. Cross-modal attention for multi-modal image registration. Med Image Anal. Nov. 2022;82:102612. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.media.2022.102612\u003c/span\u003e\u003cspan address=\"10.1016/j.media.2022.102612\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu T, Hu Y, Gao J, Sun Y, Yin B. Hierarchical Multi-Modal Transformer for Cross-Modal Long Document Classification. IEEE Trans Multimedia. 2025;27:8981\u0026ndash;94. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/TMM.2025.3608295\u003c/span\u003e\u003cspan address=\"10.1109/TMM.2025.3608295\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJin Y, Peng J, Lin X, Yuan H, Wang L, Zheng C. Multimodal transformers are hierarchical modal-wise heterogeneous graphs, in \u003cem\u003eProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)\u003c/em\u003e, 2025, pp. 2188\u0026ndash;2209. Accessed: Feb. 21, 2026. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://aclanthology.org/2025.acl-long.109/\u003c/span\u003e\u003cspan address=\"https://aclanthology.org/2025.acl-long.109/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBaltrusaitis T, Ahuja C, Morency L-P. Multimodal Machine Learning: A Survey and Taxonomy, \u003cem\u003eIEEE Trans. Pattern Anal. Mach. Intell.\u003c/em\u003e, vol. 41, no. 2, pp. 423\u0026ndash;443, Feb. 2019, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/TPAMI.2018.2798607\u003c/span\u003e\u003cspan address=\"10.1109/TPAMI.2018.2798607\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang C, Liang Z, Yan D, Hu Z, Wu T. Hgtfm: Hierarchical gating-driven transformer fusion model for robust multimodal sentiment analysis, \u003cem\u003eIEEE Access\u003c/em\u003e, 2025, Accessed: Feb. 21, 2026. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ieeexplore.ieee.org/abstract/document/10965686/\u003c/span\u003e\u003cspan address=\"https://ieeexplore.ieee.org/abstract/document/10965686/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJiao T, Guo C, Feng X, Chen Y, Song J. A Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications. CMC. 2024;80(1):1\u0026ndash;35. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.32604/cmc.2024.053204\u003c/span\u003e\u003cspan address=\"10.32604/cmc.2024.053204\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDing X, Han T, Fang Y, Larson E. An Approach for Combining Multimodal Fusion and Neural Architecture Search Applied to Knowledge Tracing, Nov. 08, 2021, \u003cem\u003earXiv\u003c/em\u003e: arXiv:2111.04497. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.48550/arXiv.2111.04497\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2111.04497\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu D, Zhang Y, Zhang J, Li Q, Zhang C, Yin Y. Multiple features fusion attention mechanism enhanced deep knowledge tracing for student performance prediction. IEEE Access. 2020;8:194894\u0026ndash;903. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/ACCESS.2020.3033200\u003c/span\u003e\u003cspan address=\"10.1109/ACCESS.2020.3033200\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePraveen Kumar B, Kalpana AV, Nalini S. Gated Attention Based Deep Learning Model for Analyzing the Influence of Social Media on Education, \u003cem\u003eJournal of Experimental \u0026amp; Theoretical Artificial Intelligence\u003c/em\u003e, vol. 37, no. 2, pp. 291\u0026ndash;305, Feb. 2025, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1080/0952813X.2023.2188262\u003c/span\u003e\u003cspan address=\"10.1080/0952813X.2023.2188262\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang R, Chi X, Zhang W, Liu G, Wang D, Wang F. Unimodal training-multimodal prediction: Cross-modal federated learning with hierarchical aggregation, \u003cem\u003eIEEE Transactions on Mobile Computing\u003c/em\u003e, 2025, Accessed: Feb. 21, 2026. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ieeexplore.ieee.org/abstract/document/10989558/\u003c/span\u003e\u003cspan address=\"https://ieeexplore.ieee.org/abstract/document/10989558/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMostafaei SH, Tanha J, Sharafkhaneh A. A novel deep learning model based on transformer and cross modality attention for classification of sleep stages. J Biomed Inform. 2024;157:104689.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePandey S, Karypis G. A Self-Attentive model for Knowledge Tracing, July 16, 2019, \u003cem\u003earXiv\u003c/em\u003e: arXiv:1907.06837. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.48550/arXiv.1907.06837\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.1907.06837\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbdelrahman G, Abdelfattah S, Wang Q, Lin Y. DBE-KT22: A Knowledge Tracing Dataset Based on Online Student Evaluation, Jan. 28, 2023, \u003cem\u003earXiv\u003c/em\u003e: arXiv:2208.12651. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.48550/arXiv.2208.12651\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2208.12651\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu Z et al. XES3G5M: A Knowledge Tracing Benchmark Dataset with Auxiliary Information.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":false,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"discover-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"diedu","sideBox":"Learn more about [Discover Education](https://www.springer.com/journal/44217)","snPcode":"44217","submissionUrl":"https://submission.nature.com/new-submission/44217/3","title":"Discover Education","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Multimodal knowledge tracing, personalized learning, affective computing in education, learning analytics, resource-constrained educational environments","lastPublishedDoi":"10.21203/rs.3.rs-9232899/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9232899/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003ePersonalized learning remains a critical challenge in education, particularly in resource-constrained environments such as Zambia, where instructional practices often follow a one-size-fits-all approach that overlooks differences in learners\u0026rsquo; cognitive processes, behaviours, and emotional states. In STEAM education, this limitation contributes to learner disengagement and poor academic performance. Although knowledge tracing (KT) techniques provide data-driven methods for modeling student learning, most existing approaches focus primarily on cognitive outcomes and fail to capture the multimodal nature of the learning process. This study proposes a multimodal approach to knowledge tracing for personalized learning that integrates cognitive and affective data to provide a more comprehensive representation of learner behaviour. The proposed framework incorporates five modalities: knowledge mastery estimates, programming interaction features, textual help-seeking patterns, affective states, and behavioural indicators. These heterogeneous data sources are combined using a context-aware deep learning architecture with hierarchical fusion and adaptive gating, enabling dynamic weighting of modalities and robustness to missing data-an important requirement in resource-constrained settings. The model was evaluated on three benchmark datasets (ASSISTments 2015, CSEDM 2019, and XES3G5M) using five-fold cross-validation on 1,065 student interaction sequences. The results demonstrate strong predictive performance, achieving a mean AUC-ROC of 0.9933\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0006, substantially outperforming traditional Bayesian Knowledge Tracing and standard Deep Knowledge Tracing models. Analysis of modality contributions shows that affective states (43.7%) and behavioural indicators (26.3%) are the most influential predictors of learning outcomes, providing empirical evidence of the importance of non-cognitive factors in personalized learning. These findings highlight the value of multimodal knowledge tracing in enhancing personalization and improving learner modelling, particularly in environments with limited educational resources. The proposed approach offers a scalable and context-aware solution for intelligent tutoring systems aimed at supporting inclusive and effective STEAM education in higher education contexts.\u003c/p\u003e","manuscriptTitle":"A Multimodal Approach to Knowledge Tracing for Personalized Learning based on Cognitive and Affective Data in Resource-Constrained Environments","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-05-15 15:32:46","doi":"10.21203/rs.3.rs-9232899/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorInvitedReview","content":"","date":"2026-05-17T13:01:51+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"268928364714696790564957425190258631638","date":"2026-05-07T10:23:24+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-05-06T15:58:31+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-04-03T15:30:17+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-04-01T17:07:25+00:00","index":"","fulltext":""},{"type":"submitted","content":"Discover Education","date":"2026-04-01T17:01:56+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"discover-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"diedu","sideBox":"Learn more about [Discover Education](https://www.springer.com/journal/44217)","snPcode":"44217","submissionUrl":"https://submission.nature.com/new-submission/44217/3","title":"Discover Education","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"c15f7415-5662-4051-b2b6-c1273643b434","owner":[],"postedDate":"May 15th, 2026","published":true,"recentEditorialEvents":[{"type":"editorInvitedReview","content":"","date":"2026-05-17T13:01:51+00:00","index":37,"fulltext":""},{"type":"reviewerAgreed","content":"268928364714696790564957425190258631638","date":"2026-05-07T10:23:24+00:00","index":34,"fulltext":""},{"type":"reviewersInvited","content":"15","date":"2026-05-06T15:58:31+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-05-15T15:32:46+00:00","versionOfRecord":[],"versionCreatedAt":"2026-05-15 15:32:46","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9232899","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9232899","identity":"rs-9232899","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}