Research on Automatic Recognition Method of Inclusive Education Classroom Behavior Based on Pose Estimation and Multimodal Fusion

Research on Automatic Recognition Method of Inclusive Education Classroom Behavior Based on Pose Estimation and Multimodal Fusion | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Research on Automatic Recognition Method of Inclusive Education Classroom Behavior Based on Pose Estimation and Multimodal Fusion Li Chuang, Li Cunhao, Zhang Yunlong, Chen Zhi, Pan Yanyan This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8205635/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 11 You are reading this latest preprint version Abstract A core challenge of inclusive education lies in the difficulty teachers face in effectively identifying and responding to the diverse behavioral manifestations of students with special educational needs (SEN) in mainstream classrooms. Traditional observation methods are time-consuming, labor-intensive, and highly subjective. This paper explores the application potential of computer vision (CV) technology in this field, aiming to construct an objective and automated framework for classroom behavior analysis. We propose a multimodal fusion method based on pose estimation and spatiotemporal modeling, capturing students' nonverbal behaviors in the classroom (such as body posture, head orientation, and activity level) using an RGB camera and combining this with simple audio features (vocal activity) for comprehensive analysis. We collected a dataset containing behavioral patterns of typical SEN students (such as Autism Spectrum Disorder (ASD) and Attention Deficit Hyperactivity Disorder (ADHD)) in a simulated inclusive education classroom environment and validated the proposed method. Experimental results show that the system achieves high accuracy (average 85.2%) in identifying key behavioral indicators (such as "attention," "social interaction," and "abnormal behavior"), significantly outperforming baseline methods that rely solely on manual observation. This study demonstrates the effectiveness of computer vision technology as a professional support tool for teachers, providing a new technological approach for achieving precise and personalized inclusive educational interventions. Physical sciences/Mathematics and computing Biological sciences/Psychology Social science/Psychology Computer vision Inclusive education Classroom behavior analysis Posture estimation Special education needs Artificial intelligence Figures Figure 1 Figure 2 Figure 3 1 Introduction Inclusive education is committed to ensuring that all students, including those with special educational needs, have equal opportunities to learn and develop fully within the general education environment[ 1 ].However, in practice, ordinary class teachers face huge challenges: they need to pay attention to the learning status of dozens of students at the same time, and provide timely and appropriate support to SEN students who may show social communication difficulties (such as autism spectrum disorder ASD) [ 2 ], inattention or hyperactivity and impulsivity (such as attention deficit hyperactivity disorder ADHD) [ 3 ] and other behavioral characteristics.Currently, the assessment of SEN students’ classroom behavior mainly relies on teachers’ subjective observations, behavioral checklists, or structured observations conducted by professionals [ 4 ].These methods have obvious limitations: first, they are discontinuous and cannot provide continuous and comprehensive behavioral data; second, they are highly dependent on the experience and subjective judgment of the observer, and reliability and validity are difficult to guarantee;Finally, for teachers, continued high-intensity observation will distract their teaching energy and form a contradiction between "observation and teaching"[ 5 ]. In recent years, the rapid development of computer vision technology, especially breakthroughs in the fields of human posture estimation [ 6 ] and behavior recognition [ 7 ], has brought hope to solve the above problems.By deploying non-intrusive ordinary RGB cameras in classrooms, the CV system can capture and quantify students' non-verbal behavioral cues, such as body posture, gestures, head movements, range of activities, etc., in real-time and objectively. These objective data can provide teachers with a "second pair of eyes" to help them identify the potential needs of SEN students earlier and more accurately, so as to implement more targeted teaching strategies [ 8 ]. Therefore, the research goal of this paper is to design and verify a classroom behavior analysis system based on computer vision, specifically serving inclusive education scenarios. Our core contributions are: (1) defining a set of key behavioral indicator systems suitable for inclusive education classrooms; (2) proposing a lightweight behavior recognition algorithm that combines posture estimation and timing modeling; (3) through empirical research, verifying the effectiveness of this technology in improving the objectivity of behavioral observations and supporting teaching decisions. 2 Research review 2.1 Key behavioral indicators in inclusive education In inclusive classrooms, the behavioral performance of SEN students is highly heterogeneous, but some common behavioral indicators are crucial for teaching intervention[ 9 ];First of all, attention concentration reflects the degree of students' investment in teaching content, and its explicit characteristics such as head facing the teacher or blackboard, stable body posture, etc. [ 10 ] provide an objective basis for judging cognitive participation; secondly, the level of social interaction is measured by indicators such as interpersonal distance, face-to-face duration, and action synchrony. It reveals students’ actual abilities in collaboration and communication [ 11 ]; and abnormal behavior patterns—such as the stereotyped repetitive movements of ASD students or the excessive activities of ADHD students—are not “problem behaviors”, but the manifestation of specific neurodevelopmental characteristics [ 12 ], which need to be functionally interpreted in conjunction with the situation.These three types of indicators together form a behavioral analysis framework that takes into account teaching practicality and individual differences (see Table 1 ), providing a quantifiable and interpretable basis for subsequent automated identification and educational intervention. Table 1 Key behavioral indicators and their observable characteristics in inclusive education Key behavioral indicators Definition Typical observable indicators Attention Engagement The extent to which students direct sensory and cognitive resources to current learning tasks -Whether the head is facing the teacher or the blackboard -Upper body posture stability (such as jitter variance, frequent shaking) Social Interaction Students’willingness and ability to communicate with peers or teachers -Euclidean distance to neighboring peers -Proportion of time spent in face-to-face interaction -Synchrony of body movements (e.g. imitation, joint steering) Atypical Behaviors Atypical behaviors specific to certain special educational needs (SEN) groups -Stereotyped repetitive movements (such as Stimming in students with ASD) -Excessive activity or restlessness (such as Hyperactivity in students with ADHD) -Be away from mission situations for long periods of time 2.2 Application of computer vision in educational behavior analysis CV technology has begun to be used in the education field. Early studies mostly used facial expression recognition to infer students' emotional states [ 13 ]. However, facial expressions are easily affected by culture and individual differences, and in real classroom environments, students often lower their heads when writing or are blocked, so the robustness is poor.In contrast, Skeleton-based Pose Estimation technology, such as OpenPose [ 6 ] and MediaPipe [ 14 ], characterizes body posture by detecting key points of the human body (such as head, shoulders, elbows, hips, knees, etc.). It is more robust and privacy-friendly (no clear face is required), and is more suitable for long-term, large-scale classroom monitoring [ 15 ].At the level of behavior recognition, researchers usually regard gesture sequences as a special spatiotemporal signal. Commonly used methods include using long short-term memory network (LSTM) [ 16 ] or spatiotemporal graph convolutional network (ST-GCN) [ 17 ] to model the dynamic changes of key points over time to classify different behavior categories.However, existing work mostly focuses on general classroom behaviors (such as raising hands, standing, walking), and lacks refined modeling of specific behaviors of SEN students. Therefore, combining computer vision technology, especially skeleton-based pose estimation with advanced temporal modeling methods, can not only improve the robustness and scalability of educational behavior analysis, but also provide new research directions and practical possibilities for future personalized behavior recognition and support systems for students with special educational needs (SEN). 2.3 The necessity of multi-modal fusion A single visual modality may suffer from insufficient information. For example, it is difficult to distinguish whether a student is thinking seriously (still) or in a daze (still) based on posture alone.The audio modality is introduced as a supplement, and by analyzing the voice activity (Voice Activity Detection, VAD) [ 18 ], it can assist in determining whether students are in active verbal interaction.This lightweight multi-modal fusion strategy can significantly improve the accuracy of behavioral understanding without excessively increasing system complexity and privacy risks [ 19 ]. Therefore, combining the complementary advantages of visual and audio modalities not only enhances the system's ability to identify students' behavioral states, but also provides feasible technical support for building a smarter and more humane educational interactive environment. 3 Research methods 3.1 System architecture In the process of promoting inclusive education, how to accurately identify students' classroom behavior has become a key challenge. The classroom behavior analysis system based on computer vision proposed in this article collects audio and video data by deploying ordinary RGB cameras and microphones, and combines MediaPipe Pose posture estimation and WebRTC voice activity detection technology to extract multi-dimensional behavioral characteristics such as student attention, social interaction and activity levels in real time, and uses the LSTM network to model the time series characteristics and automatically identify three core indicators of "attention status", "social participation" and "abnormal behavior".The system is finally visually presented on the teacher's end with a heat map and early warning mechanism, providing teachers with objective and timely decision-making support, and helping to achieve personalized and precise integrated education while ensuring privacy. The architecture of the classroom behavior analysis system proposed in this study is shown in Fig. 1 . 3.2 Data set construction Since real SEN student classroom data involves complex ethical and privacy issues, we use simulated scenarios to construct the data set. We recruited 20 actors (10 played ordinary students, 10 played SEN students, 5 of them simulated ASD characteristics, 5 simulated ADHD characteristics), and recorded "lessons" according to a preset script in a laboratory arranged as a standard primary school classroom. The scripts cover a variety of typical teaching scenarios (teacher lectures, group discussions, independent assignments) and target behaviors. All behavioral labels were independently annotated by two educational experts based on the video recordings. All experimental methods used in this study strictly adhered to relevant international guidelines (the Declaration of Helsinki) and current Chinese laws and regulations, and were carried out in accordance with corresponding technical specifications. Furthermore, the experimental protocol has been formally approved by the Biomedical Research Ethics Subcommittee of Henan University (Approval No.: HUSOM2025-978). All participants in this study and their legal guardians have fully understood the research objectives, procedures, and potential risks, and have signed written informed consent forms on a voluntary basis. 3.3 Evaluation indicators and baseline model In intelligent education behavior analysis research, a scientific evaluation system is an important basis for verifying the effectiveness of the model. This article uses four indicators: accuracy, precision, recall and F1-score to conduct a comprehensive performance evaluation on three types of tasks: attention, social interaction and abnormal behavior.In order to comprehensively measure the advantages of the proposed method, a multi-level baseline model was set up: including the theoretical lower limit Random Guess, the traditional machine learning model Pose-Only (SVM) that only relies on posture features, and the Human Observer human benchmark obtained by independently labeling 100 classroom clips by 5 experienced teachers. It not only reflects the comparison at the technical level, but also introduces the perspective of educational practice, thus more objectively reflecting the usability and advancement of the system in real teaching scenarios. 4 Experimental results and analysis 4.1 Overall performance comparison Table 2 shows the performance comparison between the proposed complete model Ours (Pose + Audio + LSTM) and each baseline model on three core behavioral indicators. The results show that the Ours model achieved optimal performance on all indicators, with an average F1-score of 85.2%, far exceeding random guessing and the SVM model using only posture features. It is particularly noteworthy that the F1-score of our model in identifying "abnormal behavior" is as high as 88.7%, which shows that the system can effectively capture the specific behavioral patterns of ASD and ADHD students. This result not only verifies the effectiveness of the multi-modal fusion strategy, but also provides reliable technical support for intelligent behavioral analysis of students with special educational needs. Table 2 Performance comparison of different models on key behavioral indicator identification tasks (F1-score %) Behavioral indicators Random Guess Pose-Only (SVM) Human Observer Ours (Pose + Audio + LSTM) Attention Engagement 33.3 76.4 72.1 84.5 Social Interaction 50.0 78.2 68.9 86.3 Atypical Behaviors 50.0 81.5 75.6 88.7 Average 44.4 78.7 72.2 85.2 4.2 Ablation experiment: the value of multi-modal fusion In order to verify the effectiveness of introducing audio modality, an ablation experiment was performed, and the results are shown (see Table 3 ). When audio features are removed, the model's performance on both indicators of "attention concentration" and "social interaction level" drops significantly (3.2 and 2.8 percentage points respectively), while the impact on "abnormal behavior patterns" is smaller. This suggests that speech information is critical for determining whether a student is actively participating (listening or speaking), but is of limited use for identifying non-verbal stereotyped movements. Further analysis shows that the audio modality not only captures whether students are speaking, but also indirectly reflects their cognitive engagement and emotional state through acoustic cues such as the rhythm, intonation, and response delay of the speech. For example, in group discussions, even if students do not take the initiative to speak, their timely modal particles (such as "um", "right") or feedback sounds while listening can be recognized by the model as signals of active participation.In contrast, "abnormal behavioral patterns" rely mainly on repetitive body movements or postural abnormalities in the visual channel and therefore rely less on audio information. This finding provides an empirical basis for multi-modal fusion strategies: when designing intelligent systems for classroom behavior analysis, different modalities should be targeted and weighted according to specific task goals to achieve an optimal balance between performance and efficiency. Table 3 Ablation experiment results(F1-score %) Model variants Attention Engagement Social Interaction Atypical Behaviors Average Complete model(Ours) 84.5 86.3 88.7 85.2 w/o Audio 81.3 83.5 88.1 84.3 4.3 Comparative analysis with human observers The radar chart shown in Fig. 3 contains four axes, representing "attention concentration," "social interaction level," "abnormal behavior pattern," and "average score." The graph area of the "Ours" model is significantly larger than that of the "Human Observer" graph, especially in the two dimensions of "social interaction level" and "abnormal behavior pattern". This suggests that automated systems surpass human observers in objectivity and consistency, particularly in identifying subtle or atypical abnormal behaviors. This advantage may stem from the model's ability to continuously and fatigue-free process multi-channel behavioral signals and capture behavioral patterns that are difficult to detect with the human eye based on large amounts of training data. For example, in a high-density classroom environment, it is often difficult for teachers to pay attention to the non-verbal cues of all students at the same time. However, the system can analyze each student's posture, facial orientation, voice interaction and other characteristics in parallel to more comprehensively assess their social participation status.In addition, the outstanding performance in the "abnormal behavior pattern" dimension also reflects the model's high sensitivity to autism spectrum or attention disorder-related behaviors such as repetitive movements and avoidance of eye contact. In contrast, human observation is susceptible to subjective experience, cognitive load, and observation blind spots, resulting in large fluctuations in ratings. Therefore, the results not only verify the effectiveness of the proposed method, but also further support the feasibility and necessity of using automated behavior analysis systems as a supplementary tool for teachers' professional judgment. 4.4 Time series dynamic visualization of key behavioral indicators The temporal dynamics of behavioral indicators shown in Fig. 4 clearly reveal the differences in adaptability of a simulated ASD student in different teaching situations. During the teacher-led lecture phase (0–5 minutes), his attention was maintained at a high level, indicating that a structured, low-social load teaching environment was more suitable for the student’s cognitive characteristics; however, after entering the group discussion phase (5–10 minutes), his attention dropped rapidly, social interaction continued to be sluggish, and abnormal behaviors (such as hand flapping) appeared frequently, reflecting that open-ended, highly interactive activities may cause anxiety or sensory overload.Notably, social interaction was only briefly elevated when the teacher directly called on students, suggesting that external guidance was a key trigger for engagement. This dynamic model not only verifies the sensitivity and explanatory power of behavioral indicators, but also provides empirical evidence for personalized teaching intervention - for example, embedding clear task instructions, visual prompts or assistance from others in group activities can effectively reduce uncertainty and improve the quality of classroom participation of this type of students. This dynamic model not only verifies the sensitivity and explanatory power of the constructed behavioral index system, but also highlights the unique value of multimodal behavioral analysis in capturing individual differentiated responses. More importantly, it provides precise and actionable empirical evidence for personalized teaching intervention. In the future, such adaptive teaching strategies based on real-time behavioral feedback are expected to achieve a substantial transformation from "unified teaching" to "teaching in accordance with aptitude" through human-machine collaboration. 5 Discussion This study confirms the great application value of computer vision technology in behavioral analysis of inclusive education classrooms. By objectively and continuously quantifying students' non-verbal behavior, this technology can effectively make up for the blind spots and limitations of teacher observation and provide timely and accurate data support for teaching intervention. However, we must be clearly aware of the ethical boundaries.First, privacy protection is the primary principle of system design and deployment. In order to maximize the protection of students' rights and interests, edge computing architecture should be adopted to ensure that the original video data is only processed locally and not transmitted externally; the system only uploads anonymized behavioral characteristic data to avoid the leakage of any personally identifiable information [ 20 ].Secondly, the role of this technology should be positioned as an "auxiliary decision-making tool" rather than a substitute for teachers' professional judgment. The system's early warning should be regarded as a situational reminder. Teachers need to comprehensively judge whether to take intervention measures based on classroom reality, individual student differences and educational experience, so as to ensure the humanism and professionalism of educational decision-making [ 21 ].Finally, algorithmic bias is a potential risk that cannot be ignored. In order to improve the fairness and universality of the model, the training data must cover student groups of different genders, races, body types, and cultural backgrounds as much as possible to prevent misjudgment or discrimination against specific groups due to data bias, and truly realize educational equity and inclusion empowered by technology [ 22 ]. In summary, the application of computer vision technology in inclusive education has broad prospects, but its implementation must be based on ethics first and people-oriented. Only on the premise of fully respecting student privacy, adhering to educational professionalism, and guarding against algorithmic bias can technology truly become a powerful support for promoting educational equity and improving teaching quality, rather than bringing new risks and injustices. Future research and practice should continue to explore the collaborative path between technology and educational ethics, and promote the development of intelligent education in a safer, fairer, and warmer direction. Future research directions : (1) Explore unsupervised or weakly supervised learning methods; (2) Correlate the behavioral analysis results with academic data to explore the causal relationship between behavioral patterns and learning outcomes; (3) Develop a student-oriented feedback interface to help SEN students self-aware and regulate their behavior. 6 Conclusion This paper successfully constructs and verifies a computer vision-based behavioral analysis framework for inclusive education classrooms. By fusing pose estimation with audio features, and utilizing an LSTM network for timing modeling, the system can automatically identify key behavioral indicators of SEN students, including attention, social interaction, and abnormal behaviors, with high accuracy.Experimental results show that this technology not only surpasses traditional manual observation methods in performance, but also provides teachers with real-time and accurate teaching support through intuitive visualization. Under the premise of strictly abiding by ethical norms, this type of smart technology is expected to become a key enabling tool to promote inclusive education from concept to in-depth practice, so that every student can be "seen", understood, and supported in the classroom. Declarations AUTHOR CONTRIBUTIONS STATEMENT Li Chuang: Conceptualization, Methodology, Formal analysis, Writing–original draft. Li Cunhao: Software, Data curation, Investigation, Validation, Visualization. Zhang Yunlong: Resources, Project administration, Supervision. Chen Zhi: Conceptualization, Methodology, Supervision, Writing review editing, Project administration. Pan Yanyan: Investigation, Data curation, Writing – review & editing, Validation. All authors have read and approved the final manuscript. Competing interests The author(s) declare no competing interests. Funding Statement The authors declare that no funding was received for this work. Data Availability Statement All data generated or analysed during this study are included in this published article . References Ainscow, M. Promoting inclusion and equity in education: lessons from international experiences[J]. Nordic J. Stud. educational policy . 6 (1), 7–16 (2020). Edition, F. Diagnostic and statistical manual of mental disorders[J]. Am. Psychiatric Assoc. 21 (21), 591–643 (2013). Polanczyk, G. V. et al. ADHD prevalence estimates across three decades: an updated systematic review and meta-regression analysis[J]. Int. J. Epidemiol. 43 (2), 434–442 (2014). Alberto, P., Troutman, A. C. & Axe, J. B. Applied behavior analysis for teachers[M] (Pearson, 2013). Florian, L. & Black-Hawkins, K. Exploring inclusive pedagogy[J]. Br. Edu. Res. J. 37 (5), 813–828 (2011). Cao, Z. et al. Openpose: Realtime multi-person 2d pose estimation using part affinity fields[J]. IEEE Trans. Pattern Anal. Mach. Intell. 43 (1), 172–186 (2019). Wang, L. et al. Temporal segment networks: Towards good practices for deep action recognition[C]//European conference on computer vision. Cham: Springer International Publishing, : 20–36. (2016). Sethi, S. S. & Jain, K. AI technologies for social emotional learning: recent research and future directions[J]. J. Res. Innovative Teach. Learn. 17 (2), 213–225 (2024). Beukelman, D. & Light, J. Augmentative and alternative communication: Supporting children and adults with complex communication needs[J]. (2020). Whitehill, J. et al. The faces of engagement: Automatic recognition of student engagementfrom facial expressions. IEEE Trans. Affect. Comput. 5 (1), 86–98 (2014). Delaherche, E. et al. Interpersonal synchrony: A survey of evaluation methods across disciplines. IEEE Trans. Affect. Comput. 3 (3), 349–365 (2012). EDITION, F. I. F. T. H. Diagnostic and statistical manual of mental disorders 205–224 ( American Psychiatric Association, 1980). D'mello, S. K. & Kory, J. A review and meta-analysis of multimodal affect detection systems[J]. ACM Comput. Surv. (CSUR) . 47 (3), 1–36 (2015). Lugaresi, C. et al. Mediapipe: A framework for perceiving and processing reality[C]//Third workshop on computer vision for AR/VR at IEEE computer vision and pattern recognition (CVPR). 2019. (2019). Whitehill, J. et al. The faces of engagement: Automatic recognition of student engagementfrom facial expressions[J]. IEEE Trans. Affect. Comput. 5 (1), 86–98 (2014). Computation, N. Long short-term memory[J]. Neural Comput. 9 , 1735–1780 (2016). Yan, S., Xiong, Y. & Lin, D. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the AAAI conference on artificial intelligence. 32(1). (2018). Sohn, J., Kim, N. S. & Sung, W. A statistical model-based voice activity detection[J]. IEEE. Signal. Process. Lett. 6 (1), 1–3 (1999). Baltrušaitis, T., Ahuja, C. & Morency, L. P. Multimodal machine learning: A survey and taxonomy[J]. IEEE Trans. Pattern Anal. Mach. Intell. 41 (2), 423–443 (2018). Mittelstadt, B. D. et al. The ethics of algorithms: Mapping the debate[J] Vol. 3, 2053951716679679 (Big Data & Society, 2016). 2. Selwyn, N. Should robots replace teachers? AI and the future of education[M] (Wiley, 2019). Buolamwini, J. & Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification[C]//Conference on fairness, accountability and transparency. PMLR, : 77–91. (2018). Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 12 Jan, 2026 Reviews received at journal 09 Jan, 2026 Reviews received at journal 02 Jan, 2026 Reviewers agreed at journal 29 Dec, 2025 Reviewers agreed at journal 27 Dec, 2025 Reviewers agreed at journal 24 Dec, 2025 Reviewers invited by journal 24 Dec, 2025 Editor assigned by journal 22 Dec, 2025 Editor invited by journal 04 Dec, 2025 Submission checks completed at journal 28 Nov, 2025 First submitted to journal 28 Nov, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8205635","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":565335063,"identity":"787517e6-8492-4f7c-bc78-bca0c85527c8","order_by":0,"name":"Li Chuang","email":"","orcid":"","institution":"Al-Farabi Kazakh National University","correspondingAuthor":false,"prefix":"","firstName":"Li","middleName":"","lastName":"Chuang","suffix":""},{"id":565335066,"identity":"abf12ef9-8f84-49fc-935c-3911f6c16e45","order_by":1,"name":"Li Cunhao","email":"","orcid":"","institution":"AL-Farabi Kazakh Nationaal University, Farabi International Business School","correspondingAuthor":false,"prefix":"","firstName":"Li","middleName":"","lastName":"Cunhao","suffix":""},{"id":565335067,"identity":"c3383b88-f5f9-416d-83b9-7aad94895f13","order_by":2,"name":"Zhang Yunlong","email":"","orcid":"","institution":"Luoyang No.1 Senior High School and No.1 Affiliated Junior High School","correspondingAuthor":false,"prefix":"","firstName":"Zhang","middleName":"","lastName":"Yunlong","suffix":""},{"id":565335068,"identity":"f278f13f-7d76-48e8-b86d-3e71a04dd4ca","order_by":3,"name":"Chen Zhi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA60lEQVRIiWNgGAWjYDACZv6HDz4Y1PDwszcfOPDBwMaOsBZ2HmbDGRXH5CR7jiUenFGQlkxYCz8PmzTPGWZjgxs+xod5PhxibCCkQ7eZ97AxbxtbYsMNBoPDNgYHmBnYDx/dgE+L2WG+xIdz22QSG2c3JBzOMbjDx8CTlnYDvxYGY4O3QFuaZQ4cAGp5xswgwWNGSIuZBG8bc2KbRGLDYQuDw4wNhLXwmEmCvM8jkcxwmIE4LWzJ4ECW4DnGcLDHIC2ZjaBfzh8+CI5K++P9nz/8+GNjx89++BheLZiAjTTlo2AUjIJRMAqwAQC6rlI/RcQloAAAAABJRU5ErkJggg==","orcid":"","institution":"Henan University","correspondingAuthor":true,"prefix":"","firstName":"Chen","middleName":"","lastName":"Zhi","suffix":""},{"id":565335069,"identity":"ec4b6025-9e99-453a-ab3c-81f37c6c3ed9","order_by":4,"name":"Pan Yanyan","email":"","orcid":"","institution":"Zhengzhou Vocational college of Cpmmerce and Tourism","correspondingAuthor":false,"prefix":"","firstName":"Pan","middleName":"","lastName":"Yanyan","suffix":""}],"badges":[],"createdAt":"2025-11-25 17:23:23","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8205635/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8205635/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":99021562,"identity":"c233e659-5425-4b0b-9b2a-e420dd82d047","added_by":"auto","created_at":"2025-12-26 05:27:15","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":542026,"visible":true,"origin":"","legend":"","description":"","filename":"ResearchonAutomaticRecognitionMethodofInclus.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205635/v1/9b6d8b31e994adcdbea2585c.docx"},{"id":99313183,"identity":"a75d267f-d709-4a8a-bd17-ab7bf461aa38","added_by":"auto","created_at":"2025-12-31 16:19:53","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6860,"visible":true,"origin":"","legend":"","description":"","filename":"165a9b2a6d2a4f67a7d58767b16db8a4.json","url":"https://assets-eu.researchsquare.com/files/rs-8205635/v1/c30d307ef48cfe8771b70fad.json"},{"id":99313178,"identity":"6caafcef-2943-480c-906a-6fd6f98ff204","added_by":"auto","created_at":"2025-12-31 16:19:52","extension":"xml","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":64600,"visible":true,"origin":"","legend":"","description":"","filename":"165a9b2a6d2a4f67a7d58767b16db8a41enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-8205635/v1/92173f7632ae1c996fcd9376.xml"},{"id":99021561,"identity":"cdb814bf-05a7-4913-bf8d-f0f67ce23e58","added_by":"auto","created_at":"2025-12-26 05:27:15","extension":"png","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":32406,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8205635/v1/0a5f5ee52a55db91610b6edf.png"},{"id":99021565,"identity":"82b1eaba-d308-4e63-90b0-579730ff66f6","added_by":"auto","created_at":"2025-12-26 05:27:15","extension":"png","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":30491,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8205635/v1/ec2ef39be60962c2ed2d63bb.png"},{"id":99313893,"identity":"4f1c44f9-b36b-4287-889c-139e63783394","added_by":"auto","created_at":"2025-12-31 16:20:34","extension":"png","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":41550,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-8205635/v1/8d6ecf238d3671a6ff52e921.png"},{"id":99313302,"identity":"8b3d89e3-5f80-45bc-a984-ff6ad86e3636","added_by":"auto","created_at":"2025-12-31 16:19:58","extension":"xml","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":61232,"visible":true,"origin":"","legend":"","description":"","filename":"165a9b2a6d2a4f67a7d58767b16db8a41structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-8205635/v1/986a54e0159c73a4dd5615f8.xml"},{"id":99021572,"identity":"2abe9bdd-854b-4338-9a74-523aa59233b9","added_by":"auto","created_at":"2025-12-26 05:27:15","extension":"html","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":73250,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-8205635/v1/67c27d0351f37075e54932e5.html"},{"id":99313477,"identity":"4060be80-9652-45f3-b91a-a8100e3a0027","added_by":"auto","created_at":"2025-12-31 16:20:13","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":179433,"visible":true,"origin":"","legend":"\u003cp\u003eArchitecture diagram of inclusive education classroom behavior analysis system based on computer vision\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8205635/v1/25f266139c31e1895b6914c5.png"},{"id":99313963,"identity":"70e8c303-c0e1-4c3c-97fb-8d0666e15dea","added_by":"auto","created_at":"2025-12-31 16:20:40","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":111813,"visible":true,"origin":"","legend":"\u003cp\u003eFigure 3 Visually compares the performance of our system with human observers on various indicators through a radar chart.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8205635/v1/0ad508291bc693e5abb62ada.png"},{"id":99313854,"identity":"312b220a-773d-4858-bc9d-279043dea441","added_by":"auto","created_at":"2025-12-31 16:20:33","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":241888,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eFigure 4 Dynamic change curve of behavioral indicators of simulated ASD students in 15 minutes of classroom activities\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-8205635/v1/ca3198b8fca7d1ceb2d9280e.png"},{"id":100406090,"identity":"76b60fae-be3c-433a-b904-034e7df51ea0","added_by":"auto","created_at":"2026-01-16 12:39:26","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1244556,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8205635/v1/e232e987-38eb-43fd-b349-09184216232d.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Research on Automatic Recognition Method of Inclusive Education Classroom Behavior Based on Pose Estimation and Multimodal Fusion","fulltext":[{"header":"1 Introduction","content":"\u003cp\u003eInclusive education is committed to ensuring that all students, including those with special educational needs, have equal opportunities to learn and develop fully within the general education environment[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e].However, in practice, ordinary class teachers face huge challenges: they need to pay attention to the learning status of dozens of students at the same time, and provide timely and appropriate support to SEN students who may show social communication difficulties (such as autism spectrum disorder ASD) [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e], inattention or hyperactivity and impulsivity (such as attention deficit hyperactivity disorder ADHD) [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e] and other behavioral characteristics.Currently, the assessment of SEN students\u0026rsquo; classroom behavior mainly relies on teachers\u0026rsquo; subjective observations, behavioral checklists, or structured observations conducted by professionals [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e].These methods have obvious limitations: first, they are discontinuous and cannot provide continuous and comprehensive behavioral data; second, they are highly dependent on the experience and subjective judgment of the observer, and reliability and validity are difficult to guarantee;Finally, for teachers, continued high-intensity observation will distract their teaching energy and form a contradiction between \"observation and teaching\"[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn recent years, the rapid development of computer vision technology, especially breakthroughs in the fields of human posture estimation [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e] and behavior recognition [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], has brought hope to solve the above problems.By deploying non-intrusive ordinary RGB cameras in classrooms, the CV system can capture and quantify students' non-verbal behavioral cues, such as body posture, gestures, head movements, range of activities, etc., in real-time and objectively. These objective data can provide teachers with a \"second pair of eyes\" to help them identify the potential needs of SEN students earlier and more accurately, so as to implement more targeted teaching strategies [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eTherefore, the research goal of this paper is to design and verify a classroom behavior analysis system based on computer vision, specifically serving inclusive education scenarios. Our core contributions are: (1) defining a set of key behavioral indicator systems suitable for inclusive education classrooms; (2) proposing a lightweight behavior recognition algorithm that combines posture estimation and timing modeling; (3) through empirical research, verifying the effectiveness of this technology in improving the objectivity of behavioral observations and supporting teaching decisions.\u003c/p\u003e"},{"header":"2 Research review","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Key behavioral indicators in inclusive education\u003c/h2\u003e \u003cp\u003eIn inclusive classrooms, the behavioral performance of SEN students is highly heterogeneous, but some common behavioral indicators are crucial for teaching intervention[\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e];First of all, attention concentration reflects the degree of students' investment in teaching content, and its explicit characteristics such as head facing the teacher or blackboard, stable body posture, etc. [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e] provide an objective basis for judging cognitive participation; secondly, the level of social interaction is measured by indicators such as interpersonal distance, face-to-face duration, and action synchrony. It reveals students\u0026rsquo; actual abilities in collaboration and communication [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]; and abnormal behavior patterns\u0026mdash;such as the stereotyped repetitive movements of ASD students or the excessive activities of ADHD students\u0026mdash;are not \u0026ldquo;problem behaviors\u0026rdquo;, but the manifestation of specific neurodevelopmental characteristics [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], which need to be functionally interpreted in conjunction with the situation.These three types of indicators together form a behavioral analysis framework that takes into account teaching practicality and individual differences (see Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e), providing a quantifiable and interpretable basis for subsequent automated identification and educational intervention.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eKey behavioral indicators and their observable characteristics in inclusive education\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eKey behavioral indicators\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDefinition\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTypical observable indicators\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAttention Engagement\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThe extent to which students direct sensory and cognitive resources to current learning tasks\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-Whether the head is facing the teacher or the blackboard\u003c/p\u003e \u003cp\u003e-Upper body posture stability (such as jitter variance, frequent shaking)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSocial Interaction\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStudents\u0026rsquo;willingness and ability to communicate with peers or teachers\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-Euclidean distance to neighboring peers\u003c/p\u003e \u003cp\u003e-Proportion of time spent in face-to-face interaction\u003c/p\u003e \u003cp\u003e-Synchrony of body movements (e.g. imitation, joint steering)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAtypical Behaviors\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAtypical behaviors specific to certain special educational needs (SEN) groups\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-Stereotyped repetitive movements (such as Stimming in students with ASD)\u003c/p\u003e \u003cp\u003e-Excessive activity or restlessness (such as Hyperactivity in students with ADHD)\u003c/p\u003e \u003cp\u003e-Be away from mission situations for long periods of time\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Application of computer vision in educational behavior analysis\u003c/h2\u003e \u003cp\u003eCV technology has begun to be used in the education field. Early studies mostly used facial expression recognition to infer students' emotional states [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. However, facial expressions are easily affected by culture and individual differences, and in real classroom environments, students often lower their heads when writing or are blocked, so the robustness is poor.In contrast, Skeleton-based Pose Estimation technology, such as OpenPose [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e] and MediaPipe [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e], characterizes body posture by detecting key points of the human body (such as head, shoulders, elbows, hips, knees, etc.). It is more robust and privacy-friendly (no clear face is required), and is more suitable for long-term, large-scale classroom monitoring [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e].At the level of behavior recognition, researchers usually regard gesture sequences as a special spatiotemporal signal. Commonly used methods include using long short-term memory network (LSTM) [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e] or spatiotemporal graph convolutional network (ST-GCN) [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e] to model the dynamic changes of key points over time to classify different behavior categories.However, existing work mostly focuses on general classroom behaviors (such as raising hands, standing, walking), and lacks refined modeling of specific behaviors of SEN students. Therefore, combining computer vision technology, especially skeleton-based pose estimation with advanced temporal modeling methods, can not only improve the robustness and scalability of educational behavior analysis, but also provide new research directions and practical possibilities for future personalized behavior recognition and support systems for students with special educational needs (SEN).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 The necessity of multi-modal fusion\u003c/h2\u003e \u003cp\u003eA single visual modality may suffer from insufficient information. For example, it is difficult to distinguish whether a student is thinking seriously (still) or in a daze (still) based on posture alone.The audio modality is introduced as a supplement, and by analyzing the voice activity (Voice Activity Detection, VAD) [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e], it can assist in determining whether students are in active verbal interaction.This lightweight multi-modal fusion strategy can significantly improve the accuracy of behavioral understanding without excessively increasing system complexity and privacy risks [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. Therefore, combining the complementary advantages of visual and audio modalities not only enhances the system's ability to identify students' behavioral states, but also provides feasible technical support for building a smarter and more humane educational interactive environment.\u003c/p\u003e \u003c/div\u003e"},{"header":"3 Research methods","content":"\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e3.1 System architecture\u003c/h2\u003e \u003cp\u003eIn the process of promoting inclusive education, how to accurately identify students' classroom behavior has become a key challenge. The classroom behavior analysis system based on computer vision proposed in this article collects audio and video data by deploying ordinary RGB cameras and microphones, and combines MediaPipe Pose posture estimation and WebRTC voice activity detection technology to extract multi-dimensional behavioral characteristics such as student attention, social interaction and activity levels in real time, and uses the LSTM network to model the time series characteristics and automatically identify three core indicators of \"attention status\", \"social participation\" and \"abnormal behavior\".The system is finally visually presented on the teacher's end with a heat map and early warning mechanism, providing teachers with objective and timely decision-making support, and helping to achieve personalized and precise integrated education while ensuring privacy. The architecture of the classroom behavior analysis system proposed in this study is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Data set construction\u003c/h2\u003e \u003cp\u003eSince real SEN student classroom data involves complex ethical and privacy issues, we use simulated scenarios to construct the data set. We recruited 20 actors (10 played ordinary students, 10 played SEN students, 5 of them simulated ASD characteristics, 5 simulated ADHD characteristics), and recorded \"lessons\" according to a preset script in a laboratory arranged as a standard primary school classroom. The scripts cover a variety of typical teaching scenarios (teacher lectures, group discussions, independent assignments) and target behaviors. All behavioral labels were independently annotated by two educational experts based on the video recordings.\u003c/p\u003e \u003cp\u003e All experimental methods used in this study strictly adhered to relevant international guidelines (the Declaration of Helsinki) and current Chinese laws and regulations, and were carried out in accordance with corresponding technical specifications. Furthermore, the experimental protocol has been formally approved by the Biomedical Research Ethics Subcommittee of Henan University (Approval No.: HUSOM2025-978). All participants in this study and their legal guardians have fully understood the research objectives, procedures, and potential risks, and have signed written informed consent forms on a voluntary basis.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Evaluation indicators and baseline model\u003c/h2\u003e \u003cp\u003eIn intelligent education behavior analysis research, a scientific evaluation system is an important basis for verifying the effectiveness of the model. This article uses four indicators: accuracy, precision, recall and F1-score to conduct a comprehensive performance evaluation on three types of tasks: attention, social interaction and abnormal behavior.In order to comprehensively measure the advantages of the proposed method, a multi-level baseline model was set up: including the theoretical lower limit Random Guess, the traditional machine learning model Pose-Only (SVM) that only relies on posture features, and the Human Observer human benchmark obtained by independently labeling 100 classroom clips by 5 experienced teachers. It not only reflects the comparison at the technical level, but also introduces the perspective of educational practice, thus more objectively reflecting the usability and advancement of the system in real teaching scenarios.\u003c/p\u003e \u003c/div\u003e"},{"header":"4 Experimental results and analysis","content":"\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Overall performance comparison\u003c/h2\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e shows the performance comparison between the proposed complete model Ours (Pose\u0026thinsp;+\u0026thinsp;Audio\u0026thinsp;+\u0026thinsp;LSTM) and each baseline model on three core behavioral indicators. The results show that the Ours model achieved optimal performance on all indicators, with an average F1-score of 85.2%, far exceeding random guessing and the SVM model using only posture features. It is particularly noteworthy that the F1-score of our model in identifying \"abnormal behavior\" is as high as 88.7%, which shows that the system can effectively capture the specific behavioral patterns of ASD and ADHD students. This result not only verifies the effectiveness of the multi-modal fusion strategy, but also provides reliable technical support for intelligent behavioral analysis of students with special educational needs.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePerformance comparison of different models on key behavioral indicator identification tasks (F1-score %)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBehavioral indicators\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRandom Guess\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePose-Only (SVM)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHuman Observer\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eOurs (Pose\u0026thinsp;+\u0026thinsp;Audio\u0026thinsp;+\u0026thinsp;LSTM)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAttention Engagement\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e33.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e76.4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e72.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e84.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSocial Interaction\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e50.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e78.2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e68.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e86.3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAtypical Behaviors\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e50.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e81.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e75.6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e88.7\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAverage\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e44.4\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e78.7\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e72.2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e85.2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Ablation experiment: the value of multi-modal fusion\u003c/h2\u003e \u003cp\u003eIn order to verify the effectiveness of introducing audio modality, an ablation experiment was performed, and the results are shown (see Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). When audio features are removed, the model's performance on both indicators of \"attention concentration\" and \"social interaction level\" drops significantly (3.2 and 2.8 percentage points respectively), while the impact on \"abnormal behavior patterns\" is smaller. This suggests that speech information is critical for determining whether a student is actively participating (listening or speaking), but is of limited use for identifying non-verbal stereotyped movements.\u003c/p\u003e \u003cp\u003eFurther analysis shows that the audio modality not only captures whether students are speaking, but also indirectly reflects their cognitive engagement and emotional state through acoustic cues such as the rhythm, intonation, and response delay of the speech. For example, in group discussions, even if students do not take the initiative to speak, their timely modal particles (such as \"um\", \"right\") or feedback sounds while listening can be recognized by the model as signals of active participation.In contrast, \"abnormal behavioral patterns\" rely mainly on repetitive body movements or postural abnormalities in the visual channel and therefore rely less on audio information. This finding provides an empirical basis for multi-modal fusion strategies: when designing intelligent systems for classroom behavior analysis, different modalities should be targeted and weighted according to specific task goals to achieve an optimal balance between performance and efficiency.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eAblation experiment results(F1-score %)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel variants\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAttention Engagement\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSocial Interaction\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAtypical Behaviors\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eAverage\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eComplete model(Ours)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e84.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e86.3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e88.7\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e85.2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ew/o Audio\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e81.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e83.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e88.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e84.3\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e4.3 Comparative analysis with human observers\u003c/h2\u003e \u003cp\u003eThe radar chart shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e3\u003c/span\u003e contains four axes, representing \"attention concentration,\" \"social interaction level,\" \"abnormal behavior pattern,\" and \"average score.\" The graph area of the \"Ours\" model is significantly larger than that of the \"Human Observer\" graph, especially in the two dimensions of \"social interaction level\" and \"abnormal behavior pattern\". This suggests that automated systems surpass human observers in objectivity and consistency, particularly in identifying subtle or atypical abnormal behaviors.\u003c/p\u003e \u003cp\u003eThis advantage may stem from the model's ability to continuously and fatigue-free process multi-channel behavioral signals and capture behavioral patterns that are difficult to detect with the human eye based on large amounts of training data. For example, in a high-density classroom environment, it is often difficult for teachers to pay attention to the non-verbal cues of all students at the same time. However, the system can analyze each student's posture, facial orientation, voice interaction and other characteristics in parallel to more comprehensively assess their social participation status.In addition, the outstanding performance in the \"abnormal behavior pattern\" dimension also reflects the model's high sensitivity to autism spectrum or attention disorder-related behaviors such as repetitive movements and avoidance of eye contact. In contrast, human observation is susceptible to subjective experience, cognitive load, and observation blind spots, resulting in large fluctuations in ratings. Therefore, the results not only verify the effectiveness of the proposed method, but also further support the feasibility and necessity of using automated behavior analysis systems as a supplementary tool for teachers' professional judgment.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003e4.4 Time series dynamic visualization of key behavioral indicators\u003c/h2\u003e \u003cp\u003eThe temporal dynamics of behavioral indicators shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e4\u003c/span\u003e clearly reveal the differences in adaptability of a simulated ASD student in different teaching situations. During the teacher-led lecture phase (0\u0026ndash;5 minutes), his attention was maintained at a high level, indicating that a structured, low-social load teaching environment was more suitable for the student\u0026rsquo;s cognitive characteristics; however, after entering the group discussion phase (5\u0026ndash;10 minutes), his attention dropped rapidly, social interaction continued to be sluggish, and abnormal behaviors (such as hand flapping) appeared frequently, reflecting that open-ended, highly interactive activities may cause anxiety or sensory overload.Notably, social interaction was only briefly elevated when the teacher directly called on students, suggesting that external guidance was a key trigger for engagement. This dynamic model not only verifies the sensitivity and explanatory power of behavioral indicators, but also provides empirical evidence for personalized teaching intervention - for example, embedding clear task instructions, visual prompts or assistance from others in group activities can effectively reduce uncertainty and improve the quality of classroom participation of this type of students.\u003c/p\u003e \u003cp\u003eThis dynamic model not only verifies the sensitivity and explanatory power of the constructed behavioral index system, but also highlights the unique value of multimodal behavioral analysis in capturing individual differentiated responses. More importantly, it provides precise and actionable empirical evidence for personalized teaching intervention. In the future, such adaptive teaching strategies based on real-time behavioral feedback are expected to achieve a substantial transformation from \"unified teaching\" to \"teaching in accordance with aptitude\" through human-machine collaboration.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"5 Discussion","content":"\u003cp\u003eThis study confirms the great application value of computer vision technology in behavioral analysis of inclusive education classrooms. By objectively and continuously quantifying students' non-verbal behavior, this technology can effectively make up for the blind spots and limitations of teacher observation and provide timely and accurate data support for teaching intervention. However, we must be clearly aware of the ethical boundaries.First, privacy protection is the primary principle of system design and deployment. In order to maximize the protection of students' rights and interests, edge computing architecture should be adopted to ensure that the original video data is only processed locally and not transmitted externally; the system only uploads anonymized behavioral characteristic data to avoid the leakage of any personally identifiable information [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e].Secondly, the role of this technology should be positioned as an \"auxiliary decision-making tool\" rather than a substitute for teachers' professional judgment. The system's early warning should be regarded as a situational reminder. Teachers need to comprehensively judge whether to take intervention measures based on classroom reality, individual student differences and educational experience, so as to ensure the humanism and professionalism of educational decision-making [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e].Finally, algorithmic bias is a potential risk that cannot be ignored. In order to improve the fairness and universality of the model, the training data must cover student groups of different genders, races, body types, and cultural backgrounds as much as possible to prevent misjudgment or discrimination against specific groups due to data bias, and truly realize educational equity and inclusion empowered by technology [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn summary, the application of computer vision technology in inclusive education has broad prospects, but its implementation must be based on ethics first and people-oriented. Only on the premise of fully respecting student privacy, adhering to educational professionalism, and guarding against algorithmic bias can technology truly become a powerful support for promoting educational equity and improving teaching quality, rather than bringing new risks and injustices. Future research and practice should continue to explore the collaborative path between technology and educational ethics, and promote the development of intelligent education in a safer, fairer, and warmer direction.\u003c/p\u003e \u003cp\u003e \u003cb\u003eFuture research directions\u003c/b\u003e:\u003c/p\u003e \u003cp\u003e(1) Explore unsupervised or weakly supervised learning methods;\u003c/p\u003e \u003cp\u003e(2) Correlate the behavioral analysis results with academic data to explore the causal relationship between behavioral patterns and learning outcomes;\u003c/p\u003e \u003cp\u003e(3) Develop a student-oriented feedback interface to help SEN students self-aware and regulate their behavior.\u003c/p\u003e"},{"header":"6 Conclusion","content":"\u003cp\u003eThis paper successfully constructs and verifies a computer vision-based behavioral analysis framework for inclusive education classrooms. By fusing pose estimation with audio features, and utilizing an LSTM network for timing modeling, the system can automatically identify key behavioral indicators of SEN students, including attention, social interaction, and abnormal behaviors, with high accuracy.Experimental results show that this technology not only surpasses traditional manual observation methods in performance, but also provides teachers with real-time and accurate teaching support through intuitive visualization. Under the premise of strictly abiding by ethical norms, this type of smart technology is expected to become a key enabling tool to promote inclusive education from concept to in-depth practice, so that every student can be \"seen\", understood, and supported in the classroom.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAUTHOR CONTRIBUTIONS STATEMENT\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eLi Chuang: Conceptualization, Methodology, Formal analysis, Writing\u0026ndash;original draft.\u003c/p\u003e\n\u003cp\u003eLi Cunhao: Software, Data curation, Investigation, Validation, Visualization.\u003c/p\u003e\n\u003cp\u003eZhang Yunlong: Resources, Project administration, Supervision.\u003c/p\u003e\n\u003cp\u003eChen Zhi: Conceptualization, Methodology, Supervision, Writing review editing, Project administration.\u003c/p\u003e\n\u003cp\u003ePan Yanyan: Investigation, Data curation, Writing\u0026nbsp;\u0026ndash;\u0026nbsp;review \u0026amp; editing, Validation.\u003c/p\u003e\n\u003cp\u003eAll authors have read and approved the final manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe author(s) declare no competing interests.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding Statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare that no funding was received for this work.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData Availability Statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll data generated or analysed during this study are included in this published article .\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eAinscow, M. Promoting inclusion and equity in education: lessons from international experiences[J]. \u003cem\u003eNordic J. Stud. educational policy\u003c/em\u003e. \u003cb\u003e6\u003c/b\u003e (1), 7\u0026ndash;16 (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEdition, F. Diagnostic and statistical manual of mental disorders[J]. \u003cem\u003eAm. Psychiatric Assoc.\u003c/em\u003e \u003cb\u003e21\u003c/b\u003e (21), 591\u0026ndash;643 (2013).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePolanczyk, G. V. et al. ADHD prevalence estimates across three decades: an updated systematic review and meta-regression analysis[J]. \u003cem\u003eInt. J. Epidemiol.\u003c/em\u003e \u003cb\u003e43\u003c/b\u003e (2), 434\u0026ndash;442 (2014).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlberto, P., Troutman, A. C. \u0026amp; Axe, J. B. \u003cem\u003eApplied behavior analysis for teachers[M]\u003c/em\u003e (Pearson, 2013).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFlorian, L. \u0026amp; Black-Hawkins, K. Exploring inclusive pedagogy[J]. \u003cem\u003eBr. Edu. Res. J.\u003c/em\u003e \u003cb\u003e37\u003c/b\u003e (5), 813\u0026ndash;828 (2011).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCao, Z. et al. Openpose: Realtime multi-person 2d pose estimation using part affinity fields[J]. \u003cem\u003eIEEE Trans. Pattern Anal. Mach. Intell.\u003c/em\u003e \u003cb\u003e43\u003c/b\u003e (1), 172\u0026ndash;186 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang, L. et al. Temporal segment networks: Towards good practices for deep action recognition[C]//European conference on computer vision. Cham: Springer International Publishing, : 20\u0026ndash;36. (2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSethi, S. S. \u0026amp; Jain, K. AI technologies for social emotional learning: recent research and future directions[J]. \u003cem\u003eJ. Res. Innovative Teach. Learn.\u003c/em\u003e \u003cb\u003e17\u003c/b\u003e (2), 213\u0026ndash;225 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBeukelman, D. \u0026amp; Light, J. Augmentative and alternative communication: Supporting children and adults with complex communication needs[J]. (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWhitehill, J. et al. The faces of engagement: Automatic recognition of student engagementfrom facial expressions. \u003cem\u003eIEEE Trans. Affect. Comput.\u003c/em\u003e \u003cb\u003e5\u003c/b\u003e (1), 86\u0026ndash;98 (2014).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDelaherche, E. et al. Interpersonal synchrony: A survey of evaluation methods across disciplines. \u003cem\u003eIEEE Trans. Affect. Comput.\u003c/em\u003e \u003cb\u003e3\u003c/b\u003e (3), 349\u0026ndash;365 (2012).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEDITION, F. I. F. T. H. \u003cem\u003eDiagnostic and statistical manual of mental disorders\u003c/em\u003e 205\u0026ndash;224 ( American Psychiatric Association, 1980).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eD'mello, S. K. \u0026amp; Kory, J. A review and meta-analysis of multimodal affect detection systems[J]. \u003cem\u003eACM Comput. Surv. (CSUR)\u003c/em\u003e. \u003cb\u003e47\u003c/b\u003e (3), 1\u0026ndash;36 (2015).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLugaresi, C. et al. Mediapipe: A framework for perceiving and processing reality[C]//Third workshop on computer vision for AR/VR at IEEE computer vision and pattern recognition (CVPR). 2019. (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWhitehill, J. et al. The faces of engagement: Automatic recognition of student engagementfrom facial expressions[J]. \u003cem\u003eIEEE Trans. Affect. Comput.\u003c/em\u003e \u003cb\u003e5\u003c/b\u003e (1), 86\u0026ndash;98 (2014).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eComputation, N. Long short-term memory[J]. \u003cem\u003eNeural Comput.\u003c/em\u003e \u003cb\u003e9\u003c/b\u003e, 1735\u0026ndash;1780 (2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYan, S., Xiong, Y. \u0026amp; Lin, D. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the AAAI conference on artificial intelligence. 32(1). (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSohn, J., Kim, N. S. \u0026amp; Sung, W. A statistical model-based voice activity detection[J]. \u003cem\u003eIEEE. Signal. Process. Lett.\u003c/em\u003e \u003cb\u003e6\u003c/b\u003e (1), 1\u0026ndash;3 (1999).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBaltrušaitis, T., Ahuja, C. \u0026amp; Morency, L. P. Multimodal machine learning: A survey and taxonomy[J]. \u003cem\u003eIEEE Trans. Pattern Anal. Mach. Intell.\u003c/em\u003e \u003cb\u003e41\u003c/b\u003e (2), 423\u0026ndash;443 (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMittelstadt, B. D. et al. \u003cem\u003eThe ethics of algorithms: Mapping the debate[J]\u003c/em\u003e Vol. 3, 2053951716679679 (Big Data \u0026amp; Society, 2016). 2.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSelwyn, N. \u003cem\u003eShould robots replace teachers? AI and the future of education[M]\u003c/em\u003e (Wiley, 2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBuolamwini, J. \u0026amp; Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification[C]//Conference on fairness, accountability and transparency. PMLR, : 77\u0026ndash;91. (2018).\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Computer vision, Inclusive education, Classroom behavior analysis, Posture estimation, Special education needs, Artificial intelligence","lastPublishedDoi":"10.21203/rs.3.rs-8205635/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8205635/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eA core challenge of inclusive education lies in the difficulty teachers face in effectively identifying and responding to the diverse behavioral manifestations of students with special educational needs (SEN) in mainstream classrooms. Traditional observation methods are time-consuming, labor-intensive, and highly subjective. This paper explores the application potential of computer vision (CV) technology in this field, aiming to construct an objective and automated framework for classroom behavior analysis. We propose a multimodal fusion method based on pose estimation and spatiotemporal modeling, capturing students' nonverbal behaviors in the classroom (such as body posture, head orientation, and activity level) using an RGB camera and combining this with simple audio features (vocal activity) for comprehensive analysis. We collected a dataset containing behavioral patterns of typical SEN students (such as Autism Spectrum Disorder (ASD) and Attention Deficit Hyperactivity Disorder (ADHD)) in a simulated inclusive education classroom environment and validated the proposed method. Experimental results show that the system achieves high accuracy (average 85.2%) in identifying key behavioral indicators (such as \"attention,\" \"social interaction,\" and \"abnormal behavior\"), significantly outperforming baseline methods that rely solely on manual observation. This study demonstrates the effectiveness of computer vision technology as a professional support tool for teachers, providing a new technological approach for achieving precise and personalized inclusive educational interventions.\u003c/p\u003e","manuscriptTitle":"Research on Automatic Recognition Method of Inclusive Education Classroom Behavior Based on Pose Estimation and Multimodal Fusion","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-26 05:27:10","doi":"10.21203/rs.3.rs-8205635/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-01-12T12:25:22+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-09T13:47:34+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-02T21:55:50+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"290702048634865450440356738744153904391","date":"2025-12-29T15:47:05+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"324317928015468605588283686138859390613","date":"2025-12-27T17:48:29+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"38293019866045612781141304988479025900","date":"2025-12-24T21:40:09+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-12-24T07:52:10+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-12-22T13:38:19+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-12-04T12:30:45+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-11-28T07:01:28+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2025-11-28T06:56:39+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d2bcccf6-3e6e-4f6f-86c3-255bed7f43c3","owner":[],"postedDate":"December 26th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":60188902,"name":"Physical sciences/Mathematics and computing"},{"id":60188903,"name":"Biological sciences/Psychology"},{"id":60188904,"name":"Social science/Psychology"}],"tags":[],"updatedAt":"2026-03-30T08:09:38+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-26 05:27:10","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8205635","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8205635","identity":"rs-8205635","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}