Exploring Teacher-Student Interaction through Multimodal Large Language Models: An Empirical Investigation

Exploring Teacher-Student Interaction through Multimodal Large Language Models: An Empirical Investigation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Exploring Teacher-Student Interaction through Multimodal Large Language Models: An Empirical Investigation Guanyu Chen, Guangxin Han, Juan Niu, Juhou He This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7620633/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 06 Feb, 2026 Read the published version in Scientific Reports → Version 1 posted 10 You are reading this latest preprint version Abstract Teacher–student interaction is central to classroom learning, yet traditional observation and machine-learning approaches often remain inefficient and subjective. This study explores the use of multimodal large language models (MLLMs) for systematic analysis of classroom dynamics. We fine-tuned VisualGLM-6B on 2,380 annotated images from 30 classroom videos, covering five interaction types: guided, collaborative, questioning, independent, and exhibitive. LoRA-based fine-tuning combined with prompt engineering was employed to enhance interpretability and domain-specific accuracy. Model performance was assessed through confusion matrices, BERTScore, and expert comparisons. The fine-tuned model achieved 82% overall accuracy, performing best on guided, independent, and exhibitive interactions, while collaborative and questioning types remained challenging. Compared with expert annotation, the model provided more structured and interpretable outputs, though occasional misclassifications and hallucinations persisted. These findings demonstrate the feasibility of applying MLLMs for efficient, objective analysis of teacher–student interactions and highlight future directions such as incorporating audio inputs and larger datasets to further advance educational research methodologies. Physical sciences/Engineering Physical sciences/Mathematics and computing Multimodal Large Language Models Artificial Intelligence Teacher-Student Interaction Behavior Analysis Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Introduction Teacher-student interaction is the core element of classroom teaching. It profoundly affects students' learning experiences and is a crucial factor determining teaching outcomes.(1) By effectively and accurately analyzing teacher-student interaction behaviors, educators can not only obtain real-time feedback on students' learning statuses,(2) but also identify and address potential classroom shortcomings(3) thereby playing a vital role in enhancing teaching quality. However, conventional approaches to analyzing teacher-student interactions, such as classroom observations and manual video analysis, although providing in-depth insights for researchers and educators, are plagued by inefficiencies, subjective biases, and challenges in managing large datasets. Furthermore, classroom behavior analyses relying on traditional machine learning techniques often get limited to single behaviors or dimensions due to technological constraints, typically providing only surface-level insights without in-depth analyses.(4) In recent years, large language models have demonstrated immense potential and influence across various fields. Especially since November 2022, the launch of OpenAI's large language model ChatGPT has garnered widespread attention, being hailed as the fastest-growing consumer application in history.(6) Subsequently, with the release of advanced models like GPT-4, its multimodal understanding and analysis capabilities further spurred industry discussions. Under this wave, global tech giants and research institutions have launched their large language models, all aiming to empower them with multimodal capabilities. Examples include Microsoft's Kosmos-2,(8) Baidu's ERNIE BOT,(52) Alibaba's Qwen-VL,(10) and Tsinghua University's VisualGLM-6B.(11) These models are all exploring the boundaries of human-computer interaction to meet various scientific, commercial, and societal needs. Multimodal large language models not only process large-scale text data but can also analyze data from other modalities like images and audio.(14) Moreover, these models possess emergent capabilities and self-explanatory characteristics, providing in-depth explanations for their analyses. This offers a fresh perspective and possibility for analyzing teacher-student interactions, with anticipated outcomes being more objective, efficient, and comprehensive. In conclusion, the application of multimodal large language model technology for analyzing teacher-student interactions exhibits substantial potential and value. The aim of this study is to explore effective methods for harnessing the advantages of this technology, thereby offering a more sophisticated and efficient approach to analyzing classroom behavior. Moreover, this study seeks to empirically validate the practical implications of implementing such an approach. Methods 1.Ethics statement All classroom videos used in this study were sourced from the open-source platform Smarteducation of China ( https://basic.smartedu.cn/ ). This online platform, created by the Ministry of Education of China, is freely available to anyone. The use of videos from this platform in this study is legal and compliant. This study has also been reviewed and approved by the ethics committees of the authors' respective regions and institutions. All experimental procedures were carried out in accordance with relevant guidelines and regulations, and complied with the principles outlined in the Declaration of Helsinki. We reiterate that all videos used in this research are open source and permitted. They are sourced from legitimate national education platforms. All use of instructional videos in this research is legal and compliant. 2. Design of Multimodal Large Language Model Application Scheme Based on the analysis presented earlier, and with a comprehensive consideration of the intricacies of classroom teaching behaviors alongside the advantages of large language models, a meticulous approach is imperative to ensure the standardization, completeness, and accuracy of the analysis outcomes. In light of this, the present study adopts a scheme that synergistically employs fine-tuning of multimodal large language models coupled with Prompts. The detailed plan is as follows: First, establish a framework for analyzing teacher-student interaction behavior. Under the guidance of this framework, construct a dataset for analyzing teacher-student interaction in classroom teaching (the part framed in yellow in Fig. 1 ). Second, select an appropriate open-source multimodal large language model as the experimental model. Utilizing the constructed dataset as a foundation, employ fine-tuning techniques to adjust the large language model (the part framed in green in Fig. 1 ). Thirdly, in accordance with the interactive behaviour analysis framework and the need for classroom behaviour analysis, the construction of prompts is undertaken. The actual class data is subsequently integrated with the Prompt, resulting in a unified set of instructions. (the part framed in red in Fig. 1 ). Finally, the formulated instructions are inputted into the meticulously calibrated multimodal large language model. Use the model's analytical and comprehension capacities to produce outcomes and structure the output in accordance with predetermined criteria. This study did not involve human participants directly. All classroom teaching videos used in this research were obtained from publicly available open platforms and were used solely for scientific research purposes. The videos were anonymized and contained no personally identifiable information. In accordance with institutional and journal requirements, formal ethics approval and informed consent were therefore not required. However, the author still obtained approval from the ethics committee. 2.1 Multimodal Large Language Models Multimodal Large Language Models (MLLM) leverage powerful Large Language Models (LLM) as their core to perform multimodal tasks. They are capable of handling not only text data but also a variety of data types such as images and audio. Recent research has demonstrated that, through appropriate training, MLLM can process, understand, and generate cross-modal information. Notably, its achievements in handling both images and text have been showcased, such as writing code based on images, discerning the deep meaning of images, and performing intricate mathematical reasoning without relying on OCR technology.(36) This study focuses on the analysis of teacher-student interactions within the classroom. Since classroom images can provide us with a visual representation of teaching activities, we approached from a visual perspective and utilized MLLM to conduct detailed analysis of real classroom images, aiming to gain deeper insights into classroom behaviors. There are many open-source large language models available for image analysis, such as LLaVA(50) and MiniGPT-4.(38) However, most of these models are primarily trained on English datasets, typically have large model parameters, and are costly to deploy. As this study mainly uses Chinese for analyzing and presenting classroom behaviors, and considering factors like dataset construction, efficacy of Chinese-language models, and hardware configurations, we selected the VisualGLM-6B model as our experimental model after testing several others. It's worth noting that,VisualGLM-6B, an innovation from Tsinghua University, is an open-source multimodal large language model endorsing images, and bilingual capabilities in Chinese and English.(39) The language model is predicated on Chat GLM-6B, encompassing 6.2 billion parameters; the imagery component, cultivated via BLIP2-Qformer, engenders a nexus between the visual model and the language model, cumulating to 7.8 billion parameters for the aggregate model. Moreover, VisualGLM-6B avails three fine-tuning methodologies, namely LoRA, QLoRA, and P-tuning. It also contemplates model deployment intricacies, integrating model quantization technology, thereby facilitating users to deploy locally on consumer-grade graphics cards. 2.2 Fine-tuning of Large Language Models Large language model fine-tuning refers to the subsequent training of the model on a specific dataset, following the completion of pre-training, in order to adapt it to the specific requirements of a given task. Such fine-tuning endeavors enhance the model's efficacy on the designated tasks.(43) In the realm of multimodal large models, fine-tuning methodologies can be broadly segregated into three categories(41) : The initial category encompasses methods of augmenting additional parameters. The fundamental premise here is to amplify the existing pre-trained model by introducing extra parameters or layers, and solely training the newly incorporated parameters, with Prefix-tuning(42) being a notable technique; The secondary category is the selective method, maintaining the model structure intact but fine-tuning specific segments or parameters of the model rather than the entire model. This method affords the selection of corresponding parameters for fine-tuning based on layer type, internal structure, or other established criteria; The tertiary category is rooted in re-parametrization, wherein the central idea is to efficiently fine-tune the model employing low-rank strategies. The concept of neural networks possessing low-dimensional representations has been extensively delved into in both empirical and theoretical analyses within deep learning, furnishing a theoretical foundation for this method's implementation. This genre of method curtails the quantum of parameters for training whilst sustaining or augmenting the model's performance, and is prevalently employed in large model fine-tuning, with LoRA epitomizing such methods. The core ethos of LoRA fine-tuning technology is to emulate parameter alterations through low-rank decomposition.(43) Specifically, it utilizes a simplistic low-rank matrix decomposition to parameterize weight updates, thereby facilitating indirect training of large models with a minimal parameter quantity. Extant research demonstrates that LoRA surpasses other prevalent fine-tuning methodologies in terms of performance in large model fine-tuning.(44) In this study, predicated on the foundation of the selected model, LoRA fine-tuning technology is selected to fine-tune the multimodal large language model, envisaging the realization of intelligent analysis of teacher-student interactive behaviors from the vantage points of technical implementation performance and precision. 3. Dataset Construction The primary objective of fine-tuning a multimodal large language model is to augment the model's capacity for domain-specific analysis. Hence, the dataset construction is a pivotal step, where the quality of the dataset directly dictates the model's performance. This experiment is centered on the analysis of teacher-student interactions, and thus, establishing a robust criterion for such analysis is fundamental for dataset construction. Given the intricate nature of analyzing teacher-student behaviors, this study meticulously considers the visual characteristics of teacher-student interactions to achieve an exhaustive analysis of classroom teaching behaviors. It integrates various factors such as spatial positioning, body movements, and teaching tools. The analysis is bifurcated to examine the features of teacher and student behaviors separately, further classifying the types of interactions between them. On the foundation of basic interaction type classifications, this study employs a method combining literature review and expert review(45) to validate the content validity of interaction types. In the literature review phase, relevant classifications and definitions concerning teacher-student interactions were extracted from existing literature. During the expert review phase, five experts in the field of educational technology were enlisted to evaluate the completeness, clarity, and relevance of the classifications. The experts provided insightful modification suggestions, and based on these recommendations, this study categorized teacher-student interactions into five distinct types, as delineated in Table 1 During the phase of dataset construction, the conventional strategy predominantly employed is manual annotation, a process that tends to be both time-consuming and labor-intensive. However, with the incremental adoption of large-scale models, some researchers have commenced exploring the auxiliary potential of ChatGPT's emergent capabilities for data annotation,(46) validating a certain degree of feasibility of this method. Expanding upon this foundation, our study amalgamates the inherent visual-text capabilities of VisuGLM-6B with the robust emergent capabilities of ChatGPT. By adhering to the established classification criteria for teacher-student interaction types and leveraging real classroom teaching videos as data support, we undertake the construction of the dataset. The rudimentary workflow is elucidated in Fig. 2 . All teaching images employed in this study were sourced from authentic classroom teaching videos. Initially, these videos were converted into images through frame extraction, as depicted in Fig. 3 . The subsequent step entails the completion of textual descriptions for the images. The visual module employed by VisualGLM is BLIP2. Extant research delineates that BLIP2 is proficient in precisely identifying the content within diverse regions of an image and discerning the respective objects therein.(42) On this premise, the original VisualGLM-6B is utilized to render fundamental descriptions of the image content. A specific example is presented in Fig. 4 . As shown in the figure above, when using the original VisualGLM-6B to analyze classroom teaching behaviors, it provides a somewhat generalized description of the objects' content but lacks precision and standardization, requiring manual corrections for accuracy enhancement. During the manual correction process, researchers need to refer to the content of images and modify and supplement analysis results accordingly by deleting inaccurate information and making modifications. Specific example is illustrated in Fig. 5 . After manual correction of the image description information, the next task is to use ChatGPT to conduct an in-depth analysis of the description. This process is carried out in two steps. The first step is to assign the role of an analyst to ChatGPT and inform it of the specific classification framework standards. Research indicates that when using Chat GPT, assigning it a role identity is conducive to improving the accuracy of its output information. The second step, based on the aforementioned dialogue, is to create Prompts. Prompts consist of four parts: the Environment section clarifies the task background; the Question section sets up the questions; the Info section includes the description of the teacher-student interaction image; the Example section clarifies the output format. The constructed Prompts are input into ChatGPT, which uses its analytical capabilities to produce structured output of the analysis results. A specific example is shown in Fig. 6 . Experts manually revise the output results based on ChatGPT's initial output, ultimately producing an analysis of the classroom teaching images. The complete analysis comprises four sections. It includes the behavior characteristics of the subjects within the image, specifically teacher behavior characteristics and student behavior characteristics, image features, and interaction types. Combined with the images and prompt words, they together form the data required for the experiment. Specific example is shown in Table 2 In this study, a total of 30 high-quality primary and secondary school teaching videos were collected and used as a basis for dataset construction. All videos were sourced from open platforms and are available for scientific research. Through the aforementioned dataset construction process, a total of 2,380 images were obtained, among which there were 737 images of Guided interaction, 485 of collaborative interaction, 321 of Questioning interaction, 359 of independent interaction, and 477 of Exhibitive interaction. The distribution of the data is shown in Fig. 7 . 4. Model Fine-tuning Based on the constructed dataset, the model was fine-tuned. During the fine-tuning process, the hyperparameters are set as follows: Table 3 Hyperparameter Settings for Fine-tuning Parameter Explanation Value Image_processor Image input size 224*224 Max_source_length Maximum input sequence length 64 Max_target_length Maximum output sequence length 256 Batch_size Training data volume per batch 8 Learning_rate Learning rate 0.0001 Train_iters Maximum number of training steps 1000 Additionally, when fine-tuning the model using the LoRA method, a two-step training strategy was employed. The first step involved LoRA fine-tuning of the BLIP2 image layer, and the second step added LoRA fine-tuning with a rank of 10 to the 1st and 12th layers of the ChatGLM model. The training process is illustrated in Fig. 8 , where the loss value continued to decrease as training progressed. In this study, to facilitate subsequent experiments and validate its application in educational settings, the model was deployed after fine-tuning. We used the Streamlit library(48) to build a video processing and intelligent analysis system for teacher-student interactive behaviors, as illustrated in Fig. 9 . Results and Discussion 1. Accuracy analysis To evaluate the accuracy of outcomes derived from fine-tuning a multimodal large language model, we carefully selected a set of 100 images that were excluded during the initial training process. These images represent five distinct categories, with each category consisting of 20 representative images. Utilizing the refined multimodal large language model, we analyzed these images. Subsequently, we classified and organized the interaction types discerned from the analysis. The culminating results are depicted in the ensuing confusion matrix. From the aforementioned figure, we discerned several significant observations across five categories: directive, cooperative, interrogative, independent, and demonstrative interactions. By meticulously examining the distinct characteristics and potential misclassifications associated with each category, we deepened our understanding of the complexities inherent in teacher-student interactions. (1) Comprehensive Analysis Among the 100 sampled images, 82 were correctly classified, indicating an 82% accuracy rate for the fine-tuned model on this dataset. This rate suggests that our refined multimodal large language model demonstrates relative efficacy in the analytical examination of teacher-student interactive behaviors. However, the possibility of misclassification underscores the multifaceted nature of classroom engagements. (2) In-depth Categorization Analysis Guided Interaction: Of the 20 images labeled as "guided," the predominant theme depicted teachers guiding students, predominantly at the classroom's forefront. Major misclassifications emerged from confounding this with "questioning" and "exhibitive" interactions. Two primary factors contributed to such inaccuracies: a misinterpretation of teacher gestures—mistakenly equating specific gestures with signaling or querying—and erroneously identifying teachers as students, leading to a misreading of the exhibitive interaction. Collaborative Interaction: In images identified as "collaborative," a clear representation of collaborative endeavors among students was evident. However, misclassifications primarily resulted from confusing this category with "independent" and " exhibitive " interactions. A salient cause for such errors stems from images that only captured a subset of students, rendering the nature of their engagement ambiguous. Questioning Interaction: Here, a salient feature is the dynamic of questioning and responding between students and educators. Nonetheless, due to visual resemblances between teachers' queries and directives, particularly in suboptimal image conditions, frequent misclassifications ensued. Independent Interaction: Images within this segment accurately captured students engrossed in solitary tasks. Yet, aspects like classroom configurations or seating arrangements often led to overlaps with "collaborative" interpretations. Exhibitive Interaction: Predominantly, this category captured students' presentations. However, the inadvertent categorization into "guided interactions" arose either from teachers' engagement in students' presentations or from image quality affecting accurate participant identification. A subsequent analysis unraveled factors contributing to misclassifications. Primarily, educators' positions and postures manifested similarly across interaction types. Secondly, student interactions occasionally became indistinct due to variables such as capture angles, proximity, and illumination. Lastly, the confluence of multiple interactional elements in some instances compounded the challenge of accurate categorization. To further analyze the differences between the model's analytical results and the original analysis results, this study utilized the language-model-based evaluation metric BERTScore for assessment and analysis. This metric calculates the similarity of tokens using contextual embeddings. It converts the generated text and the original text into tokens via the Bert model, then extracts features and computes the inner product for each corresponding word in the two texts, thus constructing a similarity matrix. The similarity of the two sentences is calculated to determine Precision, Recall, and F1 Score.(52) Precision measures which parts of the generated text correctly correspond to the reference text, with high precision indicating that every part of the generated text has a high semantic match with the reference text. Recall measures which contents of the reference text are successfully captured or covered by the generated text, with high recall suggesting that the generated text covers most of the reference text's contents. In text generation tasks, high recall indicates that the content generated encompasses most of the information from the reference text. The F1 Score is the harmonic mean of precision and recall, providing a singular measure that considers both precision and recall. In evaluating text similarity, a high F1 Score generally indicates better overall performance. In this study, similarity scores between the fine-tuned model output and the original text were calculated, averaged, and normalized. Table 4 Evaluation results Interaction type Details Precision Recall F1 Guided Interaction Teacher Behavioral Characteristics 0.775 0.712 0.733 Student Behavioral Characteristics 0.733 0.783 0.707 Image Features 0.712 0.759 0.764 Collaborative Interaction Teacher Behavioral Characteristics 0.618 0.719 0.664 Student Behavioral Characteristics 0.631 0.637 0.633 Image Features 0.651 0.623 0.636 Questioning Interaction Teacher Behavioral Characteristics 0.622 0.591 0.605 Student Behavioral Characteristics 0.723 0.671 0.696 Image Features 0.701 0.695 0.698 Independent Interaction Teacher Behavioral Characteristics 0.755 0.711 0.733 Student Behavioral Characteristics 0.733 0.683 0.707 Image Features 0.712 0.659 0.685 Exhibitive Interaction Teacher Behavioral Characteristics 0.755 0.711 0.732 Student Behavioral Characteristics 0.733 0.683 0.707 Image Features 0.712 0.659 0.685 Table 4 indicates that, among all F1 scores, the description of teacher behavior characteristics in the questioning interaction type received the lowest score. The high error rate for the questioning interaction type classification reflected in Fig. 9 is likely due to the model's incorrect behavior classification analysis of teacher actions. All F1 values for the collaborative interaction type are low, reflecting the model's inaccuracy in analyzing images of cooperative interactions. This may account for the highest error rate for cooperative interaction types as shown in Fig. 9 . Table 4 also reveals that aside from the questioning and collaborative interaction types, which have F1 values below 0.65, the F1 scores for all other types are above 0.65. This is consistent with the accuracy analysis discussed earlier in the text. This demonstrates that the large language model's determination of teacher-student interaction types is based on the description of behaviors in images, providing strong evidence that multimodal large language models are capable of interpreting the reasons behind interaction type classifications. 2. Effectiveness analysis To analyze and verify the effectiveness of the fine-tuned multimodal large language model, this study first compared the differences between the model before and after fine-tuning. Table 5 Comparison of the results of fine-tuned and unfine-tuned analysis Analysis Results Before Fine-tuning Analysis Results After Fine-tuning In this image, The teacher is writing content on the blackboard , while students sit in front of tables listening. The image depicts a group of students seated in a circle inside a classroom, with a whiteboard in the background. Such a setting is typically used for instructional demonstrations or discussions about the course content. Therefore, this image can be categorized as demonstrative interaction. Teacher Behavioral Characteristics : The teacher is standing in front of the blackboard, looking at the students in front of him, with a book and a pen in his hand. Student Behavioral Characteristics : Two students stand at the front of the classroom with what looks like a prop in their hands and explain. Image Features : Two students share their thoughts and opinions at the front of the class. One student is explaining an issue, while the other students were listening carefully and the teacher was taking notes. Interaction Type : Exhibitive Interaction. The analysis of Table 5 reveals that the outputs from the pre-fine-tuned multimodal large language model substantially differ from those after fine-tuning, despite utilizing the same prompt. The descriptions generated by the primary VisualGLM-6B model mainly focus on the content of the image, occasionally providing excessive elaboration and incorporating elements not present in the source image. For example, in the analysis results before fine-tuning, the model describes a teacher writing on a board, even though this action is not depicted in the image. In contrast to this, post-fine-tuning outputs from VisualGLM-6B exhibit more logical and structured analyses of teacher-student interactions, demonstrating an ability to adhere to predefined formats. These rigorous outputs highlight evident improvements and validate how fine-tuning enhances the analytic capabilities of models regarding teacher-student interactions. This aligns with Liu's(2022a) research which suggests that minimal data can amplify potential for large language models. To further deepen the analysis and validation of the fine-tuned multimodal large language model's analytical effectiveness, this study compared the results of the fine-tuned analysis with those of expert analyses. By examining the differences between the two, the study further explores the strengths and limitations of the fine-tuned multimodal large language model. (1) A juxtaposition of the post-fine-tuned model's and experts' analyses indicates congruence in their portrayals of teacher-student behavioral characteristics, affirming the fine-tuning method's efficacy. Delving deeper into their descriptive disparities, the fine-tuned model occasionally introduces supplemental details. As highlighted in Sample 1, the model, while acknowledging the absence of the teacher's figure, also speculates the teacher's potential side-viewing of student presentations. Such nuances underscore the model's emergent capability. (2) Even after fine-tuning, the model significantly improved its ability to classify teacher-student interactions; yet, it still struggles with occasional hallucinations. These sometimes culminate in erroneous interaction type classifications, evident in Sample 2, where it discerned a guided interaction—a stark contrast to expert evaluations. The model's inherent hallucination tendencies may be the culprit. (3) The evaluation of the output reveals persistent challenges in alignment. Despite this study's manual alignment interventions, outputs occasionally deviate from set formats or stipulations. For example, Sample 3's output omits a description of teacher behavior, potentially due to the model overlooking the teacher's depiction. Underlying issues might range from data paucity to suboptimal training samples or necessitate further model optimization—a topic meriting deeper exploration. Conclusion In this study, we utilized advanced multimodal large language models, augmented with fine-tuning and prompt-driven techniques, to conduct an intelligent analysis of teacher-student interactions. We initiated our research by curating a dataset derived from genuine classroom videos. This was followed by rigorous model fine-tuning and experimental validation, resulting in an in-depth examination of teacher-student interactions. Based on the experimental outcomes, the refined multimodal large language model demonstrated superior performance in its analysis capacity. This model not only enhanced the analytical depth and breadth but also furnished a more holistic perspective on teacher-student communication dynamics. Our approach equips educational researchers with a state-of-the-art analytical tool, facilitating the precise capture of nuanced classroom changes and interaction trends. Our findings underscore the transformational shift in educational research methodologies driven by technological advancements. Traditional observational and survey-based approaches are increasingly supplanted by digitized, intelligent tools. This paradigm shift has unlocked novel research avenues for educators, enabling a more profound exploration of classroom pedagogical dynamics. However, the study is not devoid of limitations. The current multimodal models are predominantly centered on visual and textual data. Yet, for an encompassing analysis of teacher-student interactions, audio data is indispensable. Future endeavors should emphasize the inclusion of diverse modalities, encompassing audio and gesture data. Furthermore, while our model excels in certain analytic realms, its efficacy may waver in others, underscoring the ongoing need for model optimization. The integrity of data, both in terms of quality and volume, remains paramount. To optimize model efficacy, continuous dataset expansion and refinement are imperative. Education, being an intricate domain, spans emotions, motivations, cognition, and social interactions. Sole reliance on technology for educational insights is not feasible. It's pivotal to synergize technological innovations with pedagogical and psychological insights, fostering a truly interdisciplinary research paradigm. Looking ahead, with the continuous evolution and deeper integration of technological innovations in educational research, we hold an optimistic view regarding the broader applicability of multimodal large language models in pedagogical domains. From strategizing educational interventions to evaluating pedagogical outcomes, these advanced technologies will undoubtedly play an instrumental role, warranting further research. Declarations Funding: National Natural Science Foundation of China (No. 62177032), “Research on the Autonomous Training and Evaluation Model for Pre-service Teachers’ Classroom Teaching Expression Competence.” Author Contribution Guuanyu Chen was responsible for writing and revising the article, as well as for subsequent correspondence regarding submission.Guangxin Han was responsible for creating figures and tables, as well as calibrating the technical approach.Juan Niu was responsible for providing experimental design and academic insights for this study.Juhou He was the lead investigator for this study and the project leader. He oversaw and provided guidance throughout the entire research process. Data Availability All data in this study are available for reference and upon request. If you have any questions, please contact the first author, Guanyu Chen, via email. We will respond. References Nugent, T. T.: The impact of teacher -student interaction on student motivation and achievement. https://www.semanticscholar.org/paper/The-impact-of-teacher-student-interaction-on-and-Nugent/1e2e2375e394e646a53f4f1952da4e4703cac44e?sort=is-influential (2009) Paiva, F. & Nielsen, R. D. Clustering constructed responses for formative Assessment in Comprehension SEEDING. In Lecture Notes in Computer Science (pp. 686–688). https://doi.org/10.1007/978-3-319-07221-0_107 (2014) Alonzo, A. C., Kobarg, M., & Seidel, T. Pedagogical content knowledge as reflected in teacher-student interactions: Analysis of two video cases. Journal of Research in Science Teaching, 49(10), 1211–1239. https://doi.org/10.1002/tea.21055 (2012) Chen Goldberg, P., Sümer, Ö., Stürmer, K., Wagner, W., Göllner, R., Gerjets, P., Kasneci, E., & Trautwein, U. (2019). Attentive or not? toward a machine learning approach to assessing students’ visible engagement in classroom instruction. Educational Psychology Review, 33(1), 27–49. https://doi.org/10.1007/s10648-019-09514-z (2012) g, Y. Video-based Student Classroom Classroom Behavior State analysis. International Journal of Education and Humanities, 5(2), 229–233. https://doi.org/10.54097/ijeh.v5i2.2146 (2022) Grant, N., & Metz, C. A new chat bot is a code red’for Google’s search business. New York Times, (Dec 21, 2022) ,https://www.nytimes.com/2022/12/21/technology/ai-chatgpt-google-search.html (2022) Sanderson, K. GPT-4 is here: what scientists think. Nature, 615(7954), 773. https://doi.org/10.1038/d41586-023-00816-5(2023). Peng, Z. Kosmos-2: Grounding multimodal large language models to the world. arXiv.org. https://arxiv.org/abs/2306.14824 (2023) Zhang, Z. ERNIE: Enhanced Language Representation with Informative Entities. arXiv.org. https://arxiv.org/abs/1905.07129 (2019) Bai, J. Qwen-VL: a versatile Vision-Language model for understanding, localization, text reading, and beyond. arXiv.org. https://arxiv.org/abs/2308.12966(2023) Du, Z. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. arXiv.org. https://arxiv.org/abs/2103.10360 (2021) Cornelius‐White, J. H. D. Learner-Centered Teacher-Student relationships are effective: A Meta-Analysis. Review of Educational Research, 77(1), 113–143. https://doi.org/10.3102/003465430298563 (2007) Sun, Z., Yu, Z. C., & Xu, F. Analysis and improvement of classroom teaching based on artificial intelligence. In Springer eBooks (pp. 105–121). https://doi.org/10.1007/978-3-031-09687-7_7 (2022) Wei, J. Emergent abilities of large language models. arXiv.org. https://arxiv.org/abs/2206.07682-(2022) Van De Pol, J., Volman, M., & Beishuizen, J. Scaffolding in Teacher–Student Interaction: A Decade of research. Educational Psychology Review, 22(3), 271–296. https://doi.org/10.1007/s10648-010-9127-6 (2010) Yu, J., Li, Z., Liu, Z., Tian, M., & Lu, Y. A Student-Teacher Multimodal interaction analysis system for classroom observation. In Communications in computer and information science (pp. 193–199). https://doi.org/10.1007/978-3-031-36336-8_29 (2023) Roorda, D. L., Koomen, H. M., Spilt, J. L., & Oort, F. J. The influence of affective Teacher–Student relationships on students’ school engagement and achievement. Review of Educational Research, 81(4), 493–529. https://doi.org/10.3102/0034654311421793(2011) Granic, I., & Patterson, G. R. Toward a comprehensive model of antisocial development: A dynamic systems approach. Psychological Review, 113(1), 101–131. https://doi.org/10.1037/0033-295x.113.1.101 (2006). Lin, L., & Yang, S. Exploring the influence of Teacher-Student interaction on university students’ Self-Efficacy in the flipped classroom. Journal of Education and Learning, 10(2), 84. https://doi.org/10.5539/jel.v10n2p84 (2021) Hennessy, S., Calcagni, E., Leung, A., & Mercer, N. An analysis of the forms of teacher-student dialogue that are most productive for learning. Language and Education, 37(2), 186–211. https://doi.org/10.1080/09500782.2021.1956943(2021) Flanders, N. A. Analyzing teaching behavior. Addison-Wesley. (1970) Mortelmans, D. Analyzing qualitative data using NVIVO. In Springer eBooks (pp. 435–450). https://doi.org/10.1007/978-3-030-16065-4_25 (2019) Phipps, S., & Borg, S. Exploring tensions between teachers’ grammar teaching beliefs and practices. System, 37(3), 380–390. https://doi.org/10.1016/j.system.2009.03.002 (2009) Jacobs, J., Kawanaka, T., & Stigler, J. W. Integrating qualitative and quantitative approaches to the analysis of video data on classroom teaching. International Journal of Educational Research, 31(8), 717–724. https://doi.org/10.1016/s0883-0355(99)00036-1 (1999) Sharma, A., Mohanty, D., Ijaz, M. F., & Rana, A. U. H. S. Student behavioral analysis using Computer vision. In Springer eBooks (pp. 261–266). https://doi.org/10.1007/978-981-16-2911-2_28(2021) Quadir, B., Chen, N., & Isaías, P. Analyzing the educational goals, problems and techniques used in educational big data research from 2010 to 2018. Interactive Learning Environments, 30(8), 1539–1555. https://doi.org/10.1080/10494820.2020.1712427(2020) Wang, J., Liu, T., & Wang, X. Human hand gesture recognition with convolutional neural networks for K-12 double-teachers instruction mode classroom. Infrared Physics & Technology, 111, 103464. https://doi.org/10.1016/j.infrared.2020.103464 (2020) Balti, R., Hedhili, A., Chaari, W. L., & Abed, M. Hybrid analysis of the learner’s online behavior based on learning style. Education and Information Technologies, 28(10), 12465–12504. https://doi.org/10.1007/s10639-023-11595-x(2023) Aouifi, H. E., Hajji, M. E., Es-Saady, Y., & Douzi, H. Predicting learner’s performance through video sequences viewing behavior analysis using educational data-mining. Education and Information Technologies, 26(5), 5799–5814. https://doi.org/10.1007/s10639-021-10512-4 (2021) Su, M., Cui, M., & Huang, X. Multimodal Data Fusion in Learning Analytics: A Systematic review. Sensors, 20(23), 6856. https://doi.org/10.3390/s20236856 (2020) Chang, Y. A survey on evaluation of large language models. arXiv.org. https://arxiv.org/abs/2307.03109 (2023) Wei, J. Emergent abilities of large language models. arXiv.org. https://arxiv.org/abs/2206.07682(2022) Tamkin, A. Understanding the capabilities, limitations, and societal impact of large language models. arXiv.org. https://arxiv.org/abs/2102.02503 (2021) Jeon, J. H., & Lee, S. Large language models in education: A focus on the complementary relationship between human teachers and ChatGPT. Education and Information Technologies. https://doi.org/10.1007/s10639-023-11834-1(2023) Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., . . . Kasneci, G. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274(2023) Yin, S. A survey on multimodal large language models. arXiv.org. https://arxiv.org/abs/2306.13549(2023) Liu, H. Visual instruction tuning. arXiv.org. https://arxiv.org/abs/2304.08485 (2023) Zhu, D. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv.org. https://arxiv.org/abs/2304.10592(2023) Fu, C. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language models. arXiv.org. https://arxiv.org/abs/2306.13394 (2023) Hu, Z. LLM-Adapters: an adapter family for Parameter-Efficient Fine-Tuning of large language models. arXiv.org. https://arxiv.org/abs/2304.01933 (2023) Lialin, V. Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning. arXiv.org. https://arxiv.org/abs/2303.15647 (2023) Li, X. L. Prefix-Tuning: Optimizing continuous prompts for generation. arXiv.org. https://arxiv.org/abs/2101.00190(2021) Hu, E. J. LORA: Low-Rank adaptation of Large Language Models. arXiv.org. https://arxiv.org/abs/2106.09685(2021) Wang, Y. PandaLM: an automatic evaluation benchmark for LLM instruction tuning optimization. arXiv.org. https://arxiv.org/abs/2306.05087 (2023) Woo, D. J., Wang, Y., Susanto, H., & Guo, K. Understanding English as a Foreign language Students’ idea generation Strategies for creative writing with natural language Generation tools. Journal of Educational Computing Research, 073563312311759. https://doi.org/10.1177/07356331231175999Thangaratinam, S., & Redman, C. W. E. (2005). The Delphi technique. The Obstetrician & Gynaecologist, 7(2), 120–125. https://doi.org/10.1576/toag.7.2.120.27071 (2023) Mei, X. WAVCapS: a CHATGPT-Assisted Weakly-Labelled audio captioning dataset for Audio-Language multimodal research. arXiv.org. https://arxiv.org/abs/2303.17395 (2023) Li, J. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv.org. https://arxiv.org/abs/2301.12597 (2023) Khorasani, M., Abdou, M., & Fernández, J. H. Streamlit use cases. In Apress eBooks (pp. 309–361). https://doi.org/10.1007/978-1-4842-8111-6_11(2022) Yusuf, A., Noor, N. M., & Bello, S. Using multimodal learning analytics to model students’ learning behavior in animated programming classroom. Education and Information Technologies. https://doi.org/10.1007/s10639-023-12079-8 (2023) Liu, M., Ren, Y., Nyagoga, L. M., Stonier, F., Wu, Z., & Liang, Y. Future of education in the era of generative artificial intelligence: Consensus among Chinese scholars on applications of ChatGPT in schools. Future in Educational Research, 1(1), 72–101. https://doi.org/10.1002/fer3.10 (2023) Liu, H. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. arXiv.org. https://arxiv.org/abs/2205.05638(2022) Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. BERTScore: Evaluating Text Generation with BERT. arXiv (Cornell University). https://doi.org/10.48550/arxiv.1904.09675 (2019) Tables Tables 1, 2 and 6 are available in the Supplementary Files section. Additional Declarations No competing interests reported. Supplementary Files Tables.docx Cite Share Download PDF Status: Published Journal Publication published 06 Feb, 2026 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 05 Jan, 2026 Reviews received at journal 04 Jan, 2026 Reviewers agreed at journal 15 Oct, 2025 Reviews received at journal 13 Oct, 2025 Reviewers agreed at journal 10 Oct, 2025 Reviewers invited by journal 22 Sep, 2025 Editor assigned by journal 22 Sep, 2025 Editor invited by journal 22 Sep, 2025 Submission checks completed at journal 20 Sep, 2025 First submitted to journal 20 Sep, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7620633","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":523735503,"identity":"aeaafaca-a1bd-47d5-b5ce-cb4c7af2a688","order_by":0,"name":"Guanyu Chen","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAApUlEQVRIiWNgGAWjYHACNoYPUJYEkRqY2RhnkKyFmYckLbrz+489tvlzJ8/gAPPB2zwMdnkEtZgdY2Y3zm17VmxwgC3ZmochuZgYLWzSuQ2HEzcc4DGT5mE4kNhAlBaLPyAt/N9I0MLABraFjVgtyWaSvW2HE2ceZjO2nGOQTISWwwefSfwAOqzvePPDG28q7AhrQQBmEGFAvPpRMApGwSgYBXgAAAJSOJvoKO8rAAAAAElFTkSuQmCC","orcid":"","institution":"Shaanxi Normal University","correspondingAuthor":true,"prefix":"","firstName":"Guanyu","middleName":"","lastName":"Chen","suffix":""},{"id":523735504,"identity":"ebe12ae2-5793-455d-b641-997a106d62eb","order_by":1,"name":"Guangxin Han","email":"","orcid":"","institution":"Shaanxi Normal University","correspondingAuthor":false,"prefix":"","firstName":"Guangxin","middleName":"","lastName":"Han","suffix":""},{"id":523735505,"identity":"e4032e02-aa91-43e8-8944-b544f4f9c2b6","order_by":2,"name":"Juan Niu","email":"","orcid":"","institution":"Shaanxi Normal University","correspondingAuthor":false,"prefix":"","firstName":"Juan","middleName":"","lastName":"Niu","suffix":""},{"id":523735506,"identity":"d8b30328-a725-4569-b2cb-cb0b3b347231","order_by":3,"name":"Juhou He","email":"","orcid":"","institution":"Shaanxi Normal University","correspondingAuthor":false,"prefix":"","firstName":"Juhou","middleName":"","lastName":"He","suffix":""}],"badges":[],"createdAt":"2025-09-15 12:38:21","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7620633/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7620633/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-026-38626-0","type":"published","date":"2026-02-06T15:59:55+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":92945000,"identity":"6f75468d-ee15-4ec8-83f1-f595e3db3b3e","added_by":"auto","created_at":"2025-10-07 12:30:20","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1989904,"visible":true,"origin":"","legend":"","description":"","filename":"ScientficRManuscript.docx","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/c05e8d8b5fd429d50806a503.docx"},{"id":92739891,"identity":"4e70269d-3fbc-4d79-b72e-333c4e9b7b2e","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5861,"visible":true,"origin":"","legend":"","description":"","filename":"56a3a40300a24b8e9682720c138cc024.json","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/ac9f5b5b7d302bc9236b129c.json"},{"id":92740540,"identity":"a00cc1e0-deae-4b37-b20e-e6182d34be51","added_by":"auto","created_at":"2025-10-03 17:20:05","extension":"png","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":745682,"visible":true,"origin":"","legend":"","description":"","filename":"ethicsstatement.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/4e494f4811bfa495079d641f.png"},{"id":92739899,"identity":"2c6ef85f-36d0-4810-8958-b97b6fa84c3c","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"jpg","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":357602,"visible":true,"origin":"","legend":"","description":"","filename":"researchmethod.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/8c8cdcfa3cfee283ff00b512.jpg"},{"id":92739904,"identity":"8b83dc70-68b8-4d5c-9672-d0d2abb50c5d","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"xml","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":124542,"visible":true,"origin":"","legend":"","description":"","filename":"56a3a40300a24b8e9682720c138cc0241enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/4201d7dfe8e325c83e18040f.xml"},{"id":92741194,"identity":"773914ef-ccbb-43c9-a5d8-1b8cf37042dd","added_by":"auto","created_at":"2025-10-03 17:28:05","extension":"png","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":15455,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/7338852510edcbe9000d71a6.png"},{"id":92740549,"identity":"728d479c-9456-4120-aaab-2e0c49cd15f8","added_by":"auto","created_at":"2025-10-03 17:20:05","extension":"png","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":683945,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage10.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/3ebc3bf5332326d6e383eeb3.png"},{"id":92739902,"identity":"f4fc67da-c55b-4ad1-b0e9-1c6bfa6448d6","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"png","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":33787,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage11.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/5e6d870c58f715db3f71a5f2.png"},{"id":92739946,"identity":"16c59d69-be77-4bbb-8b48-7ed30695eb6f","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":32092,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage12.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/91b324c48c0ea3d0a1d57d19.png"},{"id":92739914,"identity":"4ef8646c-0f10-47fc-b550-08e3e846b4d7","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"png","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":68919,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage13.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/0e5b1bd9209918f8c7553ec5.png"},{"id":92739916,"identity":"fcad8a41-570f-41cb-a568-b34a9b3d664c","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"png","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":21108,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage14.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/6ad7c425d0dba34fd4ca3e61.png"},{"id":92739901,"identity":"b5b0cf93-c98c-4d8c-89df-8eb98ecc75a7","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"png","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":99306,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage15.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/1a5af29ea8903c0a92b19ab0.png"},{"id":92740564,"identity":"bbde8047-f557-4695-86a4-f2321a2e2387","added_by":"auto","created_at":"2025-10-03 17:20:06","extension":"jpeg","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1965263,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage16.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/962a45805999ec4d4abfbbab.jpeg"},{"id":92739910,"identity":"f0e7f7c7-8fa5-47f3-97b3-1ae83f563e28","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"jpeg","order_by":13,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":19166,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage17.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/16ff7209271d444bdadbf00d.jpeg"},{"id":92739907,"identity":"8c20d650-fa0d-4e1b-bdd3-9103e3998191","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"png","order_by":14,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":15455,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/0dd7dbf4376de517627e3d85.png"},{"id":92739931,"identity":"b961ab1d-b252-4149-8bf7-b93c021c5823","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"emf","order_by":15,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":52032,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage19.emf","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/50423776ef4cf2f976ed08db.emf"},{"id":92741333,"identity":"ef3213ba-bdbf-442f-9d77-a93dc8f3f81b","added_by":"auto","created_at":"2025-10-03 17:36:05","extension":"emf","order_by":16,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":52032,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage19.emf","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/1ec7e9cf4c7e1b10b85ae5af.emf"},{"id":92739909,"identity":"9b08582a-d465-4e48-af4b-ab886b15e810","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"png","order_by":17,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":368888,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage20.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/ae2def8d84d3dbb7be50c86c.png"},{"id":92739913,"identity":"9b7f2ede-929a-4fb5-a168-c9ed72cf76f4","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"png","order_by":18,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":185861,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage21.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/414517b4d441415d595007bf.png"},{"id":92739912,"identity":"2b7a4198-7e2c-43c5-89bb-0365bc28e872","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"png","order_by":19,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":129331,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage22.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/d94a33cd72b37b11d1cdf9b2.png"},{"id":92739938,"identity":"1a6b8193-41f8-4827-914e-43dd25d4b38f","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":20,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":157592,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage23.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/b62a9ef59028f9796a59f439.png"},{"id":92739949,"identity":"f090dfbf-0ac3-4833-9b8c-6a519725a7ae","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":21,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":17894,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage24.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/27fb8dc280ae9360322f19d8.png"},{"id":92739911,"identity":"c3d44633-5b57-404f-93b0-0b995b667bf0","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"png","order_by":22,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":52359,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage25.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/1c6e1cb834c9d8c4c1bb7c95.png"},{"id":92741200,"identity":"fd40afed-62c9-4132-afcd-1a2408ea81dd","added_by":"auto","created_at":"2025-10-03 17:28:05","extension":"png","order_by":23,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":683945,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage10.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/ba909e619c23f4e82a11aeac.png"},{"id":92740550,"identity":"1d1614de-e1ec-4b2a-bdc4-08da8f7a95de","added_by":"auto","created_at":"2025-10-03 17:20:05","extension":"png","order_by":24,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":33787,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage11.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/05aa7d18c01e755ee74d30d1.png"},{"id":92739915,"identity":"fa3cf8b1-28ab-46c4-9f56-5703545302be","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"png","order_by":25,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":368888,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage20.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/66ace59b45aaa52260724fe2.png"},{"id":92739944,"identity":"0ba072cd-673e-40a8-b84d-5a13d909deab","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":26,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":185861,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage21.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/a0c70061433789325a9d6c89.png"},{"id":92740562,"identity":"145eed18-e4e2-455a-9502-ead29f8d8207","added_by":"auto","created_at":"2025-10-03 17:20:06","extension":"png","order_by":27,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":129331,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage22.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/d16cc04818cab20964fa4834.png"},{"id":92739947,"identity":"14842dcf-b1ab-443d-ad75-13e96a689876","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":28,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":157592,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage23.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/2c9daaebde565d1e41628271.png"},{"id":92741201,"identity":"445718c6-31db-45e5-9f8d-e9f0429bf171","added_by":"auto","created_at":"2025-10-03 17:28:06","extension":"png","order_by":29,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":99306,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage15.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/265ea88a8b48c62195f95866.png"},{"id":92739930,"identity":"35bf007b-570c-4144-85c4-725bde22f727","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":30,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":17894,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage24.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/8cac187892d28b833700e094.png"},{"id":92740563,"identity":"c2217682-4d8d-41dd-91bf-a325bddf3275","added_by":"auto","created_at":"2025-10-03 17:20:06","extension":"png","order_by":31,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":52359,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage25.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/56a004de9e9f276311503f72.png"},{"id":92739917,"identity":"01d0698e-005b-4fbc-88fb-39add1ceec58","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"png","order_by":32,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":9682,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/3cba4ac7f0235e74e038c330.png"},{"id":92740569,"identity":"8a068e05-17b1-44da-bdd3-bf6f930ff8cf","added_by":"auto","created_at":"2025-10-03 17:20:07","extension":"png","order_by":33,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":91195,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage10.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/1a16e7d1903889adf3d75ac2.png"},{"id":92741202,"identity":"c77eeae8-0bda-4031-8207-137fb2e2902d","added_by":"auto","created_at":"2025-10-03 17:28:06","extension":"png","order_by":34,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":9359,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage11.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/e4e8b96e8927fa09e1d75636.png"},{"id":92741334,"identity":"1adcde90-dd34-4a96-a24a-4538f0611d47","added_by":"auto","created_at":"2025-10-03 17:36:06","extension":"png","order_by":35,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5152,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage12.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/f71d35304d7ca93b76373974.png"},{"id":92739955,"identity":"62238df1-aa00-45e5-94a1-6b30cf921165","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":36,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":13878,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage13.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/1b5b89c34296d8811976f690.png"},{"id":92740551,"identity":"c4ee0760-00dc-4146-9086-ce8a708b2398","added_by":"auto","created_at":"2025-10-03 17:20:05","extension":"png","order_by":37,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":3359,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage14.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/2301fd332673f317aed6d769.png"},{"id":92739952,"identity":"564f5675-c9da-42a8-928c-b70473d22d3e","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":38,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":13587,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage15.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/da199f845c7fe200bd547a10.png"},{"id":92740553,"identity":"f3a2dcbb-229e-4c46-89e7-70079bfdee91","added_by":"auto","created_at":"2025-10-03 17:20:06","extension":"png","order_by":39,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":310873,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage16.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/b57caad874739c0933eda378.png"},{"id":92739937,"identity":"111da204-d780-4fb7-b01f-dd634feb6fc9","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":40,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":4347,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage17.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/9006004c51ebffb6a4aeb914.png"},{"id":92741335,"identity":"d8cf2517-a200-45cb-ac41-cf582e765d45","added_by":"auto","created_at":"2025-10-03 17:36:06","extension":"png","order_by":41,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":9682,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/a3997e01f67f4140e0ba7700.png"},{"id":92740558,"identity":"11b33c9c-cf65-4435-b942-a93142000078","added_by":"auto","created_at":"2025-10-03 17:20:06","extension":"png","order_by":42,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":8372,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage19.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/5d06eced51bd50101fb83c6e.png"},{"id":92739919,"identity":"a5f09c13-07ec-4e1f-84ff-69df774a5f1b","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"png","order_by":43,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":8372,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage19.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/67dfcf15a1cda5d0db5401c3.png"},{"id":92739959,"identity":"0f4042b3-435b-4487-bfaa-7ce16e3fc296","added_by":"auto","created_at":"2025-10-03 17:12:07","extension":"png","order_by":44,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":69772,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage20.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/aeab28d17419892943e041f5.png"},{"id":92740556,"identity":"0db4733e-cc8d-48c7-8a26-8967bdfc9f2f","added_by":"auto","created_at":"2025-10-03 17:20:06","extension":"png","order_by":45,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":26085,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage21.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/8bbe952a02c8e107e0f75fe5.png"},{"id":92739941,"identity":"04aae702-8c56-4d58-83fb-7a4b4db1dd42","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":46,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":23876,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage22.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/ebf2f4799a628c5b7599bfea.png"},{"id":92740568,"identity":"6b986380-f02c-45b0-a74d-f769928d126b","added_by":"auto","created_at":"2025-10-03 17:20:06","extension":"png","order_by":47,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":49414,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage23.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/b357f3990f7a42d44ea8197e.png"},{"id":92739923,"identity":"3836e5e3-a45e-42b4-bada-e71a51b40762","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":48,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":4210,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage24.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/05ccccd60ed9ab981f6d0759.png"},{"id":92741204,"identity":"efc20e71-5dd6-4274-9c2a-4088a6b3269b","added_by":"auto","created_at":"2025-10-03 17:28:06","extension":"png","order_by":49,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":11157,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage25.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/6a4fc46978ea0dc7a8edd5d3.png"},{"id":92740561,"identity":"74547a86-f755-4ead-8790-fe7e8acddcbd","added_by":"auto","created_at":"2025-10-03 17:20:06","extension":"png","order_by":50,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":91195,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage10.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/8786ba4a529ae04bff49f762.png"},{"id":92739927,"identity":"d8ad691c-7015-4594-9132-e6b76437e007","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":51,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":9359,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage11.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/89492b9d2609982cf484bc4a.png"},{"id":92739945,"identity":"494bd84b-76f6-4bb1-9200-9e6885fe68b8","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":52,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":69772,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage20.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/2608d1bca31446e32d15e66a.png"},{"id":92739939,"identity":"39d9494f-5108-488e-9621-5ce40dff138f","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":53,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":26085,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage21.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/c1a93b5a43a18e3f1a1fe2b4.png"},{"id":92739935,"identity":"61415ded-f136-406f-8bb1-cb404a37205d","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":54,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":23876,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/07847ca589ac27c4bf204f04.png"},{"id":92739948,"identity":"79f5f101-2889-4bb7-8a9e-27c9b3fa3a40","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":55,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":49414,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/a8aad830f3673969d3eae6cf.png"},{"id":92739921,"identity":"3914dd59-9070-42eb-ab17-8fcd0b88bd25","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"png","order_by":56,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":13587,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/40e136b7d52847494e82deaa.png"},{"id":92739957,"identity":"dfff2f02-259c-4ce7-8a29-496276535656","added_by":"auto","created_at":"2025-10-03 17:12:07","extension":"png","order_by":57,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":4210,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage24.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/82611fcf3e1f5418887cbc60.png"},{"id":92741205,"identity":"a2469764-7383-49cd-a655-cece3bf0aec1","added_by":"auto","created_at":"2025-10-03 17:28:06","extension":"png","order_by":58,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":11157,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage25.png","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/d62e52decca1725afe21036a.png"},{"id":92739943,"identity":"9a618b85-c0a4-4d83-8fbf-81102683b477","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"xml","order_by":59,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":121598,"visible":true,"origin":"","legend":"","description":"","filename":"56a3a40300a24b8e9682720c138cc0241structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/d09457b6d019c81ca0a046f2.xml"},{"id":92740565,"identity":"e2710a38-036c-4851-8775-d9190d998cd0","added_by":"auto","created_at":"2025-10-03 17:20:06","extension":"html","order_by":60,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":134063,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/ef5cd9ea6d96c71a82075c77.html"},{"id":92739888,"identity":"8249cb51-4471-488b-b5a3-e2751db6137e","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":94126,"visible":true,"origin":"","legend":"\u003cp\u003eFine-tuning strategies for multimodal large language models\u003c/p\u003e","description":"","filename":"1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/738b098c96283f514fb72311.jpg"},{"id":92740539,"identity":"189c3180-604c-4566-b6aa-51555a2b3480","added_by":"auto","created_at":"2025-10-03 17:20:05","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":95599,"visible":true,"origin":"","legend":"\u003cp\u003eFlowchart of Dataset Construction\u003c/p\u003e","description":"","filename":"2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/f48243da4de4b76f2e71973a.jpg"},{"id":92739889,"identity":"0e732c3b-e835-4f57-826e-af81447734e0","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":80439,"visible":true,"origin":"","legend":"\u003cp\u003eFrame extraction from video\u003c/p\u003e","description":"","filename":"3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/f1827b58c45b6aba701b1390.jpg"},{"id":92741330,"identity":"2b60c673-daaa-4e28-9e9e-532cc0a2bc6f","added_by":"auto","created_at":"2025-10-03 17:36:05","extension":"jpg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":119009,"visible":true,"origin":"","legend":"\u003cp\u003eBasic analysis using the original VisualGLM-6B\u003c/p\u003e","description":"","filename":"4.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/4736b97fbf00e67bf3c753d7.jpg"},{"id":92739893,"identity":"8a7ba5a4-828d-4203-9848-570e2645fde3","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"jpg","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":102579,"visible":true,"origin":"","legend":"\u003cp\u003eManual correction\u003c/p\u003e","description":"","filename":"5.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/43969d96c414197df5850506.jpg"},{"id":92739926,"identity":"93fcdb13-6e79-405f-ab9b-f33c861b6c8a","added_by":"auto","created_at":"2025-10-03 17:12:06","extension":"jpg","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":226527,"visible":true,"origin":"","legend":"\u003cp\u003eInteractive behavior analysis using ChatGPT\u003c/p\u003e","description":"","filename":"6.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/2124b835b8a83c49252588f4.jpg"},{"id":92740541,"identity":"458cb5d5-29b9-426f-abcb-e71d610e8987","added_by":"auto","created_at":"2025-10-03 17:20:05","extension":"jpg","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":61360,"visible":true,"origin":"","legend":"\u003cp\u003eData Distribution\u003c/p\u003e","description":"","filename":"7.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/5871e55c37cb8c388e78eca4.jpg"},{"id":92741198,"identity":"72a97ec9-0b2c-4392-a41a-c2fc75546c1b","added_by":"auto","created_at":"2025-10-03 17:28:05","extension":"jpg","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":86410,"visible":true,"origin":"","legend":"\u003cp\u003eModel Fine-tuning Training\u003c/p\u003e","description":"","filename":"8.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/9537ea185b81e09b48a13fd6.jpg"},{"id":92740546,"identity":"dcc91043-ade6-4b8f-994b-3cb2bbe6f2bf","added_by":"auto","created_at":"2025-10-03 17:20:05","extension":"jpg","order_by":9,"title":"Figure 9","display":"","copyAsset":false,"role":"figure","size":293765,"visible":true,"origin":"","legend":"\u003cp\u003eVideo Processing and Intelligent Analysis System\u003c/p\u003e","description":"","filename":"9.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/75b86ddc9fc4949e221e9d72.jpg"},{"id":92739900,"identity":"fbe655e5-00e6-40d3-8c78-2456bd0f0bc3","added_by":"auto","created_at":"2025-10-03 17:12:05","extension":"jpg","order_by":10,"title":"Figure 10","display":"","copyAsset":false,"role":"figure","size":68601,"visible":true,"origin":"","legend":"\u003cp\u003econfusion matrix\u003c/p\u003e","description":"","filename":"10.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/045804ee65bfa83b48c1f685.jpg"},{"id":102234224,"identity":"133afd00-fd60-4fe8-b82a-d264804506b7","added_by":"auto","created_at":"2026-02-09 16:08:02","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":5834475,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/058f882c-e907-4c6c-a4e0-b6665cfed2a3.pdf"},{"id":92741767,"identity":"a9f7c69b-fd32-4b83-a62e-d6296eede69b","added_by":"auto","created_at":"2025-10-03 17:44:05","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":242690,"visible":true,"origin":"","legend":"","description":"","filename":"Tables.docx","url":"https://assets-eu.researchsquare.com/files/rs-7620633/v1/870785d81afb6bbbc5d8fe7d.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Exploring Teacher-Student Interaction through Multimodal Large Language Models: An Empirical Investigation","fulltext":[{"header":"Introduction","content":"\u003cp\u003eTeacher-student interaction is the core element of classroom teaching. It profoundly affects students' learning experiences and is a crucial factor determining teaching outcomes.(1) By effectively and accurately analyzing teacher-student interaction behaviors, educators can not only obtain real-time feedback on students' learning statuses,(2) but also identify and address potential classroom shortcomings(3) thereby playing a vital role in enhancing teaching quality. However, conventional approaches to analyzing teacher-student interactions, such as classroom observations and manual video analysis, although providing in-depth insights for researchers and educators, are plagued by inefficiencies, subjective biases, and challenges in managing large datasets. Furthermore, classroom behavior analyses relying on traditional machine learning techniques often get limited to single behaviors or dimensions due to technological constraints, typically providing only surface-level insights without in-depth analyses.(4)\u003c/p\u003e\u003cp\u003eIn recent years, large language models have demonstrated immense potential and influence across various fields. Especially since November 2022, the launch of OpenAI's large language model ChatGPT has garnered widespread attention, being hailed as the fastest-growing consumer application in history.(6) Subsequently, with the release of advanced models like GPT-4, its multimodal understanding and analysis capabilities further spurred industry discussions. Under this wave, global tech giants and research institutions have launched their large language models, all aiming to empower them with multimodal capabilities. Examples include Microsoft's Kosmos-2,(8) Baidu's ERNIE BOT,(52) Alibaba's Qwen-VL,(10) and Tsinghua University's VisualGLM-6B.(11) These models are all exploring the boundaries of human-computer interaction to meet various scientific, commercial, and societal needs.\u003c/p\u003e\u003cp\u003eMultimodal large language models not only process large-scale text data but can also analyze data from other modalities like images and audio.(14) Moreover, these models possess emergent capabilities and self-explanatory characteristics, providing in-depth explanations for their analyses. This offers a fresh perspective and possibility for analyzing teacher-student interactions, with anticipated outcomes being more objective, efficient, and comprehensive.\u003c/p\u003e\u003cp\u003eIn conclusion, the application of multimodal large language model technology for analyzing teacher-student interactions exhibits substantial potential and value. The aim of this study is to explore effective methods for harnessing the advantages of this technology, thereby offering a more sophisticated and efficient approach to analyzing classroom behavior. Moreover, this study seeks to empirically validate the practical implications of implementing such an approach.\u003c/p\u003e"},{"header":"Methods","content":"\n\u003ch3\u003e1.Ethics statement\u003c/h3\u003e\n\u003cp\u003eAll classroom videos used in this study were sourced from the open-source platform Smarteducation of China (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://basic.smartedu.cn/\u003c/span\u003e\u003cspan address=\"https://basic.smartedu.cn/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e). This online platform, created by the Ministry of Education of China, is freely available to anyone. The use of videos from this platform in this study is legal and compliant. This study has also been reviewed and approved by the ethics committees of the authors' respective regions and institutions.\u003c/p\u003e\u003cp\u003e All experimental procedures were carried out in accordance with relevant guidelines and regulations, and complied with the principles outlined in the Declaration of Helsinki.\u003c/p\u003e\u003cp\u003eWe reiterate that all videos used in this research are open source and permitted. They are sourced from legitimate national education platforms. All use of instructional videos in this research is legal and compliant.\u003c/p\u003e\n\u003ch3\u003e2. Design of Multimodal Large Language Model Application Scheme\u003c/h3\u003e\n\u003cp\u003eBased on the analysis presented earlier, and with a comprehensive consideration of the intricacies of classroom teaching behaviors alongside the advantages of large language models, a meticulous approach is imperative to ensure the standardization, completeness, and accuracy of the analysis outcomes. In light of this, the present study adopts a scheme that synergistically employs fine-tuning of multimodal large language models coupled with Prompts.\u003c/p\u003e\u003cp\u003eThe detailed plan is as follows: First, establish a framework for analyzing teacher-student interaction behavior. Under the guidance of this framework, construct a dataset for analyzing teacher-student interaction in classroom teaching (the part framed in yellow in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Second, select an appropriate open-source multimodal large language model as the experimental model. Utilizing the constructed dataset as a foundation, employ fine-tuning techniques to adjust the large language model (the part framed in green in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Thirdly, in accordance with the interactive behaviour analysis framework and the need for classroom behaviour analysis, the construction of prompts is undertaken. The actual class data is subsequently integrated with the Prompt, resulting in a unified set of instructions. (the part framed in red in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Finally, the formulated instructions are inputted into the meticulously calibrated multimodal large language model. Use the model's analytical and comprehension capacities to produce outcomes and structure the output in accordance with predetermined criteria.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eThis study did not involve human participants directly. All classroom teaching videos used in this research were obtained from publicly available open platforms and were used solely for scientific research purposes. The videos were anonymized and contained no personally identifiable information. In accordance with institutional and journal requirements, formal ethics approval and informed consent were therefore not required. However, the author still obtained approval from the ethics committee.\u003c/p\u003e\u003cdiv id=\"Sec5\" class=\"Section2\"\u003e\u003ch2\u003e2.1 Multimodal Large Language Models\u003c/h2\u003e\u003cp\u003eMultimodal Large Language Models (MLLM) leverage powerful Large Language Models (LLM) as their core to perform multimodal tasks. They are capable of handling not only text data but also a variety of data types such as images and audio. Recent research has demonstrated that, through appropriate training, MLLM can process, understand, and generate cross-modal information. Notably, its achievements in handling both images and text have been showcased, such as writing code based on images, discerning the deep meaning of images, and performing intricate mathematical reasoning without relying on OCR technology.(36)\u003c/p\u003e\u003cp\u003eThis study focuses on the analysis of teacher-student interactions within the classroom. Since classroom images can provide us with a visual representation of teaching activities, we approached from a visual perspective and utilized MLLM to conduct detailed analysis of real classroom images, aiming to gain deeper insights into classroom behaviors.\u003c/p\u003e\u003cp\u003eThere are many open-source large language models available for image analysis, such as LLaVA(50) and MiniGPT-4.(38) However, most of these models are primarily trained on English datasets, typically have large model parameters, and are costly to deploy. As this study mainly uses Chinese for analyzing and presenting classroom behaviors, and considering factors like dataset construction, efficacy of Chinese-language models, and hardware configurations, we selected the VisualGLM-6B model as our experimental model after testing several others.\u003c/p\u003e\u003cp\u003eIt's worth noting that,VisualGLM-6B, an innovation from Tsinghua University, is an open-source multimodal large language model endorsing images, and bilingual capabilities in Chinese and English.(39) The language model is predicated on Chat GLM-6B, encompassing 6.2\u0026nbsp;billion parameters; the imagery component, cultivated via BLIP2-Qformer, engenders a nexus between the visual model and the language model, cumulating to 7.8\u0026nbsp;billion parameters for the aggregate model. Moreover, VisualGLM-6B avails three fine-tuning methodologies, namely LoRA, QLoRA, and P-tuning. It also contemplates model deployment intricacies, integrating model quantization technology, thereby facilitating users to deploy locally on consumer-grade graphics cards.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec6\" class=\"Section2\"\u003e\u003ch2\u003e2.2 Fine-tuning of Large Language Models\u003c/h2\u003e\u003cp\u003eLarge language model fine-tuning refers to the subsequent training of the model on a specific dataset, following the completion of pre-training, in order to adapt it to the specific requirements of a given task. Such fine-tuning endeavors enhance the model's efficacy on the designated tasks.(43)\u003c/p\u003e\u003cp\u003eIn the realm of multimodal large models, fine-tuning methodologies can be broadly segregated into three categories(41) : The initial category encompasses methods of augmenting additional parameters. The fundamental premise here is to amplify the existing pre-trained model by introducing extra parameters or layers, and solely training the newly incorporated parameters, with Prefix-tuning(42) being a notable technique; The secondary category is the selective method, maintaining the model structure intact but fine-tuning specific segments or parameters of the model rather than the entire model. This method affords the selection of corresponding parameters for fine-tuning based on layer type, internal structure, or other established criteria; The tertiary category is rooted in re-parametrization, wherein the central idea is to efficiently fine-tune the model employing low-rank strategies. The concept of neural networks possessing low-dimensional representations has been extensively delved into in both empirical and theoretical analyses within deep learning, furnishing a theoretical foundation for this method's implementation. This genre of method curtails the quantum of parameters for training whilst sustaining or augmenting the model's performance, and is prevalently employed in large model fine-tuning, with LoRA epitomizing such methods.\u003c/p\u003e\u003cp\u003eThe core ethos of LoRA fine-tuning technology is to emulate parameter alterations through low-rank decomposition.(43) Specifically, it utilizes a simplistic low-rank matrix decomposition to parameterize weight updates, thereby facilitating indirect training of large models with a minimal parameter quantity. Extant research demonstrates that LoRA surpasses other prevalent fine-tuning methodologies in terms of performance in large model fine-tuning.(44) In this study, predicated on the foundation of the selected model, LoRA fine-tuning technology is selected to fine-tune the multimodal large language model, envisaging the realization of intelligent analysis of teacher-student interactive behaviors from the vantage points of technical implementation performance and precision.\u003c/p\u003e\u003c/div\u003e\n\u003ch3\u003e3. Dataset Construction\u003c/h3\u003e\n\u003cp\u003eThe primary objective of fine-tuning a multimodal large language model is to augment the model's capacity for domain-specific analysis. Hence, the dataset construction is a pivotal step, where the quality of the dataset directly dictates the model's performance. This experiment is centered on the analysis of teacher-student interactions, and thus, establishing a robust criterion for such analysis is fundamental for dataset construction.\u003c/p\u003e\u003cp\u003eGiven the intricate nature of analyzing teacher-student behaviors, this study meticulously considers the visual characteristics of teacher-student interactions to achieve an exhaustive analysis of classroom teaching behaviors. It integrates various factors such as spatial positioning, body movements, and teaching tools. The analysis is bifurcated to examine the features of teacher and student behaviors separately, further classifying the types of interactions between them. On the foundation of basic interaction type classifications, this study employs a method combining literature review and expert review(45) to validate the content validity of interaction types. In the literature review phase, relevant classifications and definitions concerning teacher-student interactions were extracted from existing literature. During the expert review phase, five experts in the field of educational technology were enlisted to evaluate the completeness, clarity, and relevance of the classifications. The experts provided insightful modification suggestions, and based on these recommendations, this study categorized teacher-student interactions into five distinct types, as delineated in Table\u0026nbsp;1\u003c/p\u003e\u003cp\u003eDuring the phase of dataset construction, the conventional strategy predominantly employed is manual annotation, a process that tends to be both time-consuming and labor-intensive. However, with the incremental adoption of large-scale models, some researchers have commenced exploring the auxiliary potential of ChatGPT's emergent capabilities for data annotation,(46) validating a certain degree of feasibility of this method. Expanding upon this foundation, our study amalgamates the inherent visual-text capabilities of VisuGLM-6B with the robust emergent capabilities of ChatGPT. By adhering to the established classification criteria for teacher-student interaction types and leveraging real classroom teaching videos as data support, we undertake the construction of the dataset. The rudimentary workflow is elucidated in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eAll teaching images employed in this study were sourced from authentic classroom teaching videos. Initially, these videos were converted into images through frame extraction, as depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eThe subsequent step entails the completion of textual descriptions for the images. The visual module employed by VisualGLM is BLIP2. Extant research delineates that BLIP2 is proficient in precisely identifying the content within diverse regions of an image and discerning the respective objects therein.(42) On this premise, the original VisualGLM-6B is utilized to render fundamental descriptions of the image content. A specific example is presented in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eAs shown in the figure above, when using the original VisualGLM-6B to analyze classroom teaching behaviors, it provides a somewhat generalized description of the objects' content but lacks precision and standardization, requiring manual corrections for accuracy enhancement. During the manual correction process, researchers need to refer to the content of images and modify and supplement analysis results accordingly by deleting inaccurate information and making modifications. Specific example is illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eAfter manual correction of the image description information, the next task is to use ChatGPT to conduct an in-depth analysis of the description. This process is carried out in two steps. The first step is to assign the role of an analyst to ChatGPT and inform it of the specific classification framework standards. Research indicates that when using Chat GPT, assigning it a role identity is conducive to improving the accuracy of its output information. The second step, based on the aforementioned dialogue, is to create Prompts. Prompts consist of four parts: the Environment section clarifies the task background; the Question section sets up the questions; the Info section includes the description of the teacher-student interaction image; the Example section clarifies the output format. The constructed Prompts are input into ChatGPT, which uses its analytical capabilities to produce structured output of the analysis results. A specific example is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003e.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eExperts manually revise the output results based on ChatGPT's initial output, ultimately producing an analysis of the classroom teaching images. The complete analysis comprises four sections. It includes the behavior characteristics of the subjects within the image, specifically teacher behavior characteristics and student behavior characteristics, image features, and interaction types. Combined with the images and prompt words, they together form the data required for the experiment. Specific example is shown in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e2\u003c/span\u003e\u003c/p\u003e\u003cp\u003eIn this study, a total of 30 high-quality primary and secondary school teaching videos were collected and used as a basis for dataset construction. All videos were sourced from open platforms and are available for scientific research. Through the aforementioned dataset construction process, a total of 2,380 images were obtained, among which there were 737 images of Guided interaction, 485 of collaborative interaction, 321 of Questioning interaction, 359 of independent interaction, and 477 of Exhibitive interaction. The distribution of the data is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e7\u003c/span\u003e.\u003c/p\u003e\n\u003ch3\u003e4. Model Fine-tuning\u003c/h3\u003e\n\u003cp\u003eBased on the constructed dataset, the model was fine-tuned. During the fine-tuning process, the hyperparameters are set as follows:\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eHyperparameter Settings for Fine-tuning\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"3\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eParameter\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eExplanation\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eValue\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eImage_processor\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eImage input size\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e224*224\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eMax_source_length\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMaximum input sequence length\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e64\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eMax_target_length\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMaximum output sequence length\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e256\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eBatch_size\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eTraining data volume per batch\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e8\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eLearning_rate\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eLearning rate\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.0001\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eTrain_iters\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eMaximum number of training steps\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e1000\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eAdditionally, when fine-tuning the model using the LoRA method, a two-step training strategy was employed. The first step involved LoRA fine-tuning of the BLIP2 image layer, and the second step added LoRA fine-tuning with a rank of 10 to the 1st and 12th layers of the ChatGLM model. The training process is illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig8\" class=\"InternalRef\"\u003e8\u003c/span\u003e, where the loss value continued to decrease as training progressed.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eIn this study, to facilitate subsequent experiments and validate its application in educational settings, the model was deployed after fine-tuning. We used the Streamlit library(48) to build a video processing and intelligent analysis system for teacher-student interactive behaviors, as illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig9\" class=\"InternalRef\"\u003e9\u003c/span\u003e.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e"},{"header":"Results and Discussion","content":"\u003ch3\u003e1. Accuracy analysis\u003c/h3\u003e\n\u003cp\u003eTo evaluate the accuracy of outcomes derived from fine-tuning a multimodal large language model, we carefully selected a set of 100 images that were excluded during the initial training process. These images represent five distinct categories, with each category consisting of 20 representative images. Utilizing the refined multimodal large language model, we analyzed these images. Subsequently, we classified and organized the interaction types discerned from the analysis. The culminating results are depicted in the ensuing confusion matrix.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eFrom the aforementioned figure, we discerned several significant observations across five categories: directive, cooperative, interrogative, independent, and demonstrative interactions. By meticulously examining the distinct characteristics and potential misclassifications associated with each category, we deepened our understanding of the complexities inherent in teacher-student interactions.\u003c/p\u003e\u003cp\u003e(1) Comprehensive Analysis\u003c/p\u003e\u003cp\u003eAmong the 100 sampled images, 82 were correctly classified, indicating an 82% accuracy rate for the fine-tuned model on this dataset. This rate suggests that our refined multimodal large language model demonstrates relative efficacy in the analytical examination of teacher-student interactive behaviors. However, the possibility of misclassification underscores the multifaceted nature of classroom engagements.\u003c/p\u003e\u003cp\u003e(2) In-depth Categorization Analysis\u003c/p\u003e\u003cp\u003eGuided Interaction: Of the 20 images labeled as \"guided,\" the predominant theme depicted teachers guiding students, predominantly at the classroom's forefront. Major misclassifications emerged from confounding this with \"questioning\" and \"exhibitive\" interactions. Two primary factors contributed to such inaccuracies: a misinterpretation of teacher gestures\u0026mdash;mistakenly equating specific gestures with signaling or querying\u0026mdash;and erroneously identifying teachers as students, leading to a misreading of the exhibitive interaction.\u003c/p\u003e\u003cp\u003eCollaborative Interaction: In images identified as \"collaborative,\" a clear representation of collaborative endeavors among students was evident. However, misclassifications primarily resulted from confusing this category with \"independent\" and \" exhibitive \" interactions. A salient cause for such errors stems from images that only captured a subset of students, rendering the nature of their engagement ambiguous.\u003c/p\u003e\u003cp\u003eQuestioning Interaction: Here, a salient feature is the dynamic of questioning and responding between students and educators. Nonetheless, due to visual resemblances between teachers' queries and directives, particularly in suboptimal image conditions, frequent misclassifications ensued.\u003c/p\u003e\u003cp\u003eIndependent Interaction: Images within this segment accurately captured students engrossed in solitary tasks. Yet, aspects like classroom configurations or seating arrangements often led to overlaps with \"collaborative\" interpretations.\u003c/p\u003e\u003cp\u003eExhibitive Interaction: Predominantly, this category captured students' presentations. However, the inadvertent categorization into \"guided interactions\" arose either from teachers' engagement in students' presentations or from image quality affecting accurate participant identification.\u003c/p\u003e\u003cp\u003eA subsequent analysis unraveled factors contributing to misclassifications. Primarily, educators' positions and postures manifested similarly across interaction types. Secondly, student interactions occasionally became indistinct due to variables such as capture angles, proximity, and illumination. Lastly, the confluence of multiple interactional elements in some instances compounded the challenge of accurate categorization.\u003c/p\u003e\u003cp\u003eTo further analyze the differences between the model's analytical results and the original analysis results, this study utilized the language-model-based evaluation metric BERTScore for assessment and analysis. This metric calculates the similarity of tokens using contextual embeddings. It converts the generated text and the original text into tokens via the Bert model, then extracts features and computes the inner product for each corresponding word in the two texts, thus constructing a similarity matrix. The similarity of the two sentences is calculated to determine Precision, Recall, and F1 Score.(52) Precision measures which parts of the generated text correctly correspond to the reference text, with high precision indicating that every part of the generated text has a high semantic match with the reference text. Recall measures which contents of the reference text are successfully captured or covered by the generated text, with high recall suggesting that the generated text covers most of the reference text's contents. In text generation tasks, high recall indicates that the content generated encompasses most of the information from the reference text. The F1 Score is the harmonic mean of precision and recall, providing a singular measure that considers both precision and recall. In evaluating text similarity, a high F1 Score generally indicates better overall performance. In this study, similarity scores between the fine-tuned model output and the original text were calculated, averaged, and normalized.\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eEvaluation results\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"5\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eInteraction type\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eDetails\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003ePrecision\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003eRecall\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003eF1\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e\u003cp\u003eGuided Interaction\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eTeacher Behavioral Characteristics\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.775\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.712\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.733\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eStudent Behavioral Characteristics\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.733\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.783\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.707\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eImage Features\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.712\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.759\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.764\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e\u003cp\u003eCollaborative Interaction\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eTeacher Behavioral Characteristics\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.618\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.719\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.664\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eStudent Behavioral Characteristics\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.631\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.637\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.633\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eImage Features\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.651\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.623\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.636\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e\u003cp\u003eQuestioning Interaction\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eTeacher Behavioral Characteristics\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.622\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.591\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.605\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eStudent Behavioral Characteristics\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.723\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.671\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.696\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eImage Features\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.701\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.695\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.698\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e\u003cp\u003eIndependent Interaction\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eTeacher Behavioral Characteristics\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.755\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.711\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.733\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eStudent Behavioral Characteristics\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.733\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.683\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.707\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eImage Features\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.712\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.659\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.685\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e\u003cp\u003eExhibitive Interaction\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eTeacher Behavioral Characteristics\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.755\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.711\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.732\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eStudent Behavioral Characteristics\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.733\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.683\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.707\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eImage Features\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.712\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.659\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.685\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e4\u003c/span\u003e indicates that, among all F1 scores, the description of teacher behavior characteristics in the questioning interaction type received the lowest score. The high error rate for the questioning interaction type classification reflected in Fig.\u0026nbsp;\u003cspan refid=\"Fig9\" class=\"InternalRef\"\u003e9\u003c/span\u003e is likely due to the model's incorrect behavior classification analysis of teacher actions.\u003c/p\u003e\u003cp\u003eAll F1 values for the collaborative interaction type are low, reflecting the model's inaccuracy in analyzing images of cooperative interactions. This may account for the highest error rate for cooperative interaction types as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig9\" class=\"InternalRef\"\u003e9\u003c/span\u003e.\u003c/p\u003e\u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e4\u003c/span\u003e also reveals that aside from the questioning and collaborative interaction types, which have F1 values below 0.65, the F1 scores for all other types are above 0.65. This is consistent with the accuracy analysis discussed earlier in the text. This demonstrates that the large language model's determination of teacher-student interaction types is based on the description of behaviors in images, providing strong evidence that multimodal large language models are capable of interpreting the reasons behind interaction type classifications.\u003c/p\u003e\n\u003ch3\u003e2. Effectiveness analysis\u003c/h3\u003e\n\u003cp\u003eTo analyze and verify the effectiveness of the fine-tuned multimodal large language model, this study first compared the differences between the model before and after fine-tuning.\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eComparison of the results of fine-tuned and unfine-tuned analysis\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"2\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAnalysis Results Before Fine-tuning\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eAnalysis Results After Fine-tuning\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eIn this image, \u003cb\u003eThe teacher is writing content on the blackboard\u003c/b\u003e, while students sit in front of tables listening. The image depicts a group of students seated in a circle inside a classroom, with a whiteboard in the background. Such a setting is typically used for instructional demonstrations or discussions about the course content. Therefore, this image can be categorized as demonstrative interaction.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e\u003cb\u003eTeacher Behavioral Characteristics\u003c/b\u003e: The teacher is standing in front of the blackboard, looking at the students in front of him, with a book and a pen in his hand.\u003c/p\u003e\u003cp\u003e\u003cb\u003eStudent Behavioral Characteristics\u003c/b\u003e: Two students stand at the front of the classroom with what looks like a prop in their hands and explain.\u003c/p\u003e\u003cp\u003e\u003cb\u003eImage Features\u003c/b\u003e: Two students share their thoughts and opinions at the front of the class. One student is explaining an issue, while the other students were listening carefully and the teacher was taking notes.\u003c/p\u003e\u003cp\u003e\u003cb\u003eInteraction Type\u003c/b\u003e: Exhibitive Interaction.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eThe analysis of Table \u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e5\u003c/span\u003e reveals that the outputs from the pre-fine-tuned multimodal large language model substantially differ from those after fine-tuning, despite utilizing the same prompt. The descriptions generated by the primary VisualGLM-6B model mainly focus on the content of the image, occasionally providing excessive elaboration and incorporating elements not present in the source image. For example, in the analysis results before fine-tuning, the model describes a teacher writing on a board, even though this action is not depicted in the image. In contrast to this, post-fine-tuning outputs from VisualGLM-6B exhibit more logical and structured analyses of teacher-student interactions, demonstrating an ability to adhere to predefined formats. These rigorous outputs highlight evident improvements and validate how fine-tuning enhances the analytic capabilities of models regarding teacher-student interactions. This aligns with Liu's(2022a) research which suggests that minimal data can amplify potential for large language models.\u003c/p\u003e\u003cp\u003eTo further deepen the analysis and validation of the fine-tuned multimodal large language model's analytical effectiveness, this study compared the results of the fine-tuned analysis with those of expert analyses. By examining the differences between the two, the study further explores the strengths and limitations of the fine-tuned multimodal large language model.\u003c/p\u003e\u003cp\u003e(1) A juxtaposition of the post-fine-tuned model's and experts' analyses indicates congruence in their portrayals of teacher-student behavioral characteristics, affirming the fine-tuning method's efficacy. Delving deeper into their descriptive disparities, the fine-tuned model occasionally introduces supplemental details. As highlighted in Sample 1, the model, while acknowledging the absence of the teacher's figure, also speculates the teacher's potential side-viewing of student presentations. Such nuances underscore the model's emergent capability.\u003c/p\u003e\u003cp\u003e(2) Even after fine-tuning, the model significantly improved its ability to classify teacher-student interactions; yet, it still struggles with occasional hallucinations. These sometimes culminate in erroneous interaction type classifications, evident in Sample 2, where it discerned a guided interaction\u0026mdash;a stark contrast to expert evaluations. The model's inherent hallucination tendencies may be the culprit.\u003c/p\u003e\u003cp\u003e(3) The evaluation of the output reveals persistent challenges in alignment. Despite this study's manual alignment interventions, outputs occasionally deviate from set formats or stipulations. For example, Sample 3's output omits a description of teacher behavior, potentially due to the model overlooking the teacher's depiction. Underlying issues might range from data paucity to suboptimal training samples or necessitate further model optimization\u0026mdash;a topic meriting deeper exploration.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eIn this study, we utilized advanced multimodal large language models, augmented with fine-tuning and prompt-driven techniques, to conduct an intelligent analysis of teacher-student interactions. We initiated our research by curating a dataset derived from genuine classroom videos. This was followed by rigorous model fine-tuning and experimental validation, resulting in an in-depth examination of teacher-student interactions.\u003c/p\u003e\u003cp\u003eBased on the experimental outcomes, the refined multimodal large language model demonstrated superior performance in its analysis capacity. This model not only enhanced the analytical depth and breadth but also furnished a more holistic perspective on teacher-student communication dynamics. Our approach equips educational researchers with a state-of-the-art analytical tool, facilitating the precise capture of nuanced classroom changes and interaction trends.\u003c/p\u003e\u003cp\u003eOur findings underscore the transformational shift in educational research methodologies driven by technological advancements. Traditional observational and survey-based approaches are increasingly supplanted by digitized, intelligent tools. This paradigm shift has unlocked novel research avenues for educators, enabling a more profound exploration of classroom pedagogical dynamics.\u003c/p\u003e\u003cp\u003eHowever, the study is not devoid of limitations. The current multimodal models are predominantly centered on visual and textual data. Yet, for an encompassing analysis of teacher-student interactions, audio data is indispensable. Future endeavors should emphasize the inclusion of diverse modalities, encompassing audio and gesture data.\u003c/p\u003e\u003cp\u003eFurthermore, while our model excels in certain analytic realms, its efficacy may waver in others, underscoring the ongoing need for model optimization. The integrity of data, both in terms of quality and volume, remains paramount. To optimize model efficacy, continuous dataset expansion and refinement are imperative.\u003c/p\u003e\u003cp\u003eEducation, being an intricate domain, spans emotions, motivations, cognition, and social interactions. Sole reliance on technology for educational insights is not feasible. It's pivotal to synergize technological innovations with pedagogical and psychological insights, fostering a truly interdisciplinary research paradigm.\u003c/p\u003e\u003cp\u003eLooking ahead, with the continuous evolution and deeper integration of technological innovations in educational research, we hold an optimistic view regarding the broader applicability of multimodal large language models in pedagogical domains. From strategizing educational interventions to evaluating pedagogical outcomes, these advanced technologies will undoubtedly play an instrumental role, warranting further research.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eFunding: National Natural Science Foundation of China (No. 62177032), \u0026ldquo;Research on the Autonomous Training and Evaluation Model for Pre-service Teachers\u0026rsquo; Classroom Teaching Expression Competence.\u0026rdquo;\u003c/p\u003e\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eGuuanyu Chen was responsible for writing and revising the article, as well as for subsequent correspondence regarding submission.Guangxin Han was responsible for creating figures and tables, as well as calibrating the technical approach.Juan Niu was responsible for providing experimental design and academic insights for this study.Juhou He was the lead investigator for this study and the project leader. He oversaw and provided guidance throughout the entire research process.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eAll data in this study are available for reference and upon request. If you have any questions, please contact the first author, Guanyu Chen, via email. We will respond.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eNugent, T. T.: The impact of teacher -student interaction on student motivation and achievement. https://www.semanticscholar.org/paper/The-impact-of-teacher-student-interaction-on-and-Nugent/1e2e2375e394e646a53f4f1952da4e4703cac44e?sort=is-influential (2009)\u003c/li\u003e\n\u003cli\u003ePaiva, F. \u0026amp; Nielsen, R. D. Clustering constructed responses for formative Assessment in Comprehension SEEDING. In Lecture Notes in Computer Science (pp. 686\u0026ndash;688). https://doi.org/10.1007/978-3-319-07221-0_107 (2014)\u003c/li\u003e\n\u003cli\u003eAlonzo, A. C., Kobarg, M., \u0026amp; Seidel, T. Pedagogical content knowledge as reflected in teacher-student interactions: Analysis of two video cases. Journal of Research in Science Teaching, 49(10), 1211\u0026ndash;1239. https://doi.org/10.1002/tea.21055 (2012)\u003c/li\u003e\n\u003cli\u003eChen Goldberg, P., S\u0026uuml;mer, \u0026Ouml;., St\u0026uuml;rmer, K., Wagner, W., G\u0026ouml;llner, R., Gerjets, P., Kasneci, E., \u0026amp; Trautwein, U. (2019). Attentive or not? toward a machine learning approach to assessing students\u0026rsquo; visible engagement in classroom instruction. Educational Psychology Review, 33(1), 27\u0026ndash;49. https://doi.org/10.1007/s10648-019-09514-z (2012)\u003c/li\u003e\n\u003cli\u003eg, Y. Video-based Student Classroom Classroom Behavior State analysis. International Journal of Education and Humanities, 5(2), 229\u0026ndash;233. https://doi.org/10.54097/ijeh.v5i2.2146 (2022)\u003c/li\u003e\n\u003cli\u003eGrant, N., \u0026amp; Metz, C. A new chat bot is a code red\u0026rsquo;for Google\u0026rsquo;s search business. New York Times, (Dec 21, 2022) ,https://www.nytimes.com/2022/12/21/technology/ai-chatgpt-google-search.html (2022)\u003c/li\u003e\n\u003cli\u003eSanderson, K. GPT-4 is here: what scientists think. Nature, 615(7954), 773. https://doi.org/10.1038/d41586-023-00816-5(2023). \u003c/li\u003e\n\u003cli\u003ePeng, Z. Kosmos-2: Grounding multimodal large language models to the world. arXiv.org. https://arxiv.org/abs/2306.14824 (2023)\u003c/li\u003e\n\u003cli\u003eZhang, Z. ERNIE: Enhanced Language Representation with Informative Entities. arXiv.org. https://arxiv.org/abs/1905.07129 (2019)\u003c/li\u003e\n\u003cli\u003eBai, J. Qwen-VL: a versatile Vision-Language model for understanding, localization, text reading, and beyond. arXiv.org. https://arxiv.org/abs/2308.12966(2023)\u003c/li\u003e\n\u003cli\u003eDu, Z. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. arXiv.org. https://arxiv.org/abs/2103.10360 (2021)\u003c/li\u003e\n\u003cli\u003eCornelius‐White, J. H. D. Learner-Centered Teacher-Student relationships are effective: A Meta-Analysis. Review of Educational Research, 77(1), 113\u0026ndash;143. https://doi.org/10.3102/003465430298563 (2007)\u003c/li\u003e\n\u003cli\u003eSun, Z., Yu, Z. C., \u0026amp; Xu, F. Analysis and improvement of classroom teaching based on artificial intelligence. In Springer eBooks (pp. 105\u0026ndash;121). https://doi.org/10.1007/978-3-031-09687-7_7 (2022)\u003c/li\u003e\n\u003cli\u003eWei, J. Emergent abilities of large language models. arXiv.org. https://arxiv.org/abs/2206.07682-(2022)\u003c/li\u003e\n\u003cli\u003eVan De Pol, J., Volman, M., \u0026amp; Beishuizen, J. Scaffolding in Teacher\u0026ndash;Student Interaction: A Decade of research. Educational Psychology Review, 22(3), 271\u0026ndash;296. https://doi.org/10.1007/s10648-010-9127-6 (2010)\u003c/li\u003e\n\u003cli\u003eYu, J., Li, Z., Liu, Z., Tian, M., \u0026amp; Lu, Y. A Student-Teacher Multimodal interaction analysis system for classroom observation. In Communications in computer and information science (pp. 193\u0026ndash;199). https://doi.org/10.1007/978-3-031-36336-8_29 (2023)\u003c/li\u003e\n\u003cli\u003eRoorda, D. L., Koomen, H. M., Spilt, J. L., \u0026amp; Oort, F. J. The influence of affective Teacher\u0026ndash;Student relationships on students\u0026rsquo; school engagement and achievement. Review of Educational Research, 81(4), 493\u0026ndash;529. https://doi.org/10.3102/0034654311421793(2011)\u003c/li\u003e\n\u003cli\u003eGranic, I., \u0026amp; Patterson, G. R. Toward a comprehensive model of antisocial development: A dynamic systems approach. Psychological Review, 113(1), 101\u0026ndash;131. https://doi.org/10.1037/0033-295x.113.1.101 (2006).\u003c/li\u003e\n\u003cli\u003eLin, L., \u0026amp; Yang, S. Exploring the influence of Teacher-Student interaction on university students\u0026rsquo; Self-Efficacy in the flipped classroom. Journal of Education and Learning, 10(2), 84. https://doi.org/10.5539/jel.v10n2p84 (2021)\u003c/li\u003e\n\u003cli\u003eHennessy, S., Calcagni, E., Leung, A., \u0026amp; Mercer, N. An analysis of the forms of teacher-student dialogue that are most productive for learning. Language and Education, 37(2), 186\u0026ndash;211. https://doi.org/10.1080/09500782.2021.1956943(2021)\u003c/li\u003e\n\u003cli\u003eFlanders, N. A. Analyzing teaching behavior. Addison-Wesley. (1970)\u003c/li\u003e\n\u003cli\u003eMortelmans, D. Analyzing qualitative data using NVIVO. In Springer eBooks (pp. 435\u0026ndash;450). https://doi.org/10.1007/978-3-030-16065-4_25 (2019)\u003c/li\u003e\n\u003cli\u003ePhipps, S., \u0026amp; Borg, S. Exploring tensions between teachers\u0026rsquo; grammar teaching beliefs and practices. System, 37(3), 380\u0026ndash;390. https://doi.org/10.1016/j.system.2009.03.002 (2009)\u003c/li\u003e\n\u003cli\u003eJacobs, J., Kawanaka, T., \u0026amp; Stigler, J. W. Integrating qualitative and quantitative approaches to the analysis of video data on classroom teaching. International Journal of Educational Research, 31(8), 717\u0026ndash;724. https://doi.org/10.1016/s0883-0355(99)00036-1 (1999)\u003c/li\u003e\n\u003cli\u003eSharma, A., Mohanty, D., Ijaz, M. F., \u0026amp; Rana, A. U. H. S. Student behavioral analysis using Computer vision. In Springer eBooks (pp. 261\u0026ndash;266). https://doi.org/10.1007/978-981-16-2911-2_28(2021)\u003c/li\u003e\n\u003cli\u003eQuadir, B., Chen, N., \u0026amp; Isa\u0026iacute;as, P. Analyzing the educational goals, problems and techniques used in educational big data research from 2010 to 2018. Interactive Learning Environments, 30(8), 1539\u0026ndash;1555. https://doi.org/10.1080/10494820.2020.1712427(2020)\u003c/li\u003e\n\u003cli\u003eWang, J., Liu, T., \u0026amp; Wang, X. Human hand gesture recognition with convolutional neural networks for K-12 double-teachers instruction mode classroom. Infrared Physics \u0026amp; Technology, 111, 103464. https://doi.org/10.1016/j.infrared.2020.103464 (2020)\u003c/li\u003e\n\u003cli\u003eBalti, R., Hedhili, A., Chaari, W. L., \u0026amp; Abed, M. Hybrid analysis of the learner\u0026rsquo;s online behavior based on learning style. Education and Information Technologies, 28(10), 12465\u0026ndash;12504. https://doi.org/10.1007/s10639-023-11595-x(2023)\u003c/li\u003e\n\u003cli\u003eAouifi, H. E., Hajji, M. E., Es-Saady, Y., \u0026amp; Douzi, H. Predicting learner\u0026rsquo;s performance through video sequences viewing behavior analysis using educational data-mining. Education and Information Technologies, 26(5), 5799\u0026ndash;5814. https://doi.org/10.1007/s10639-021-10512-4 (2021)\u003c/li\u003e\n\u003cli\u003eSu, M., Cui, M., \u0026amp; Huang, X. Multimodal Data Fusion in Learning Analytics: A Systematic review. Sensors, 20(23), 6856. https://doi.org/10.3390/s20236856 (2020)\u003c/li\u003e\n\u003cli\u003eChang, Y. A survey on evaluation of large language models. arXiv.org. https://arxiv.org/abs/2307.03109 (2023)\u003c/li\u003e\n\u003cli\u003eWei, J. Emergent abilities of large language models. arXiv.org. https://arxiv.org/abs/2206.07682(2022)\u003c/li\u003e\n\u003cli\u003eTamkin, A. Understanding the capabilities, limitations, and societal impact of large language models. arXiv.org. https://arxiv.org/abs/2102.02503 (2021)\u003c/li\u003e\n\u003cli\u003eJeon, J. H., \u0026amp; Lee, S. Large language models in education: A focus on the complementary relationship between human teachers and ChatGPT. Education and Information Technologies. https://doi.org/10.1007/s10639-023-11834-1(2023)\u003c/li\u003e\n\u003cli\u003eKasneci, E., Sessler, K., K\u0026uuml;chemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., G\u0026uuml;nnemann, S., H\u0026uuml;llermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., . . . Kasneci, G. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274(2023)\u003c/li\u003e\n\u003cli\u003eYin, S. A survey on multimodal large language models. arXiv.org. https://arxiv.org/abs/2306.13549(2023)\u003c/li\u003e\n\u003cli\u003eLiu, H. Visual instruction tuning. arXiv.org. https://arxiv.org/abs/2304.08485 (2023)\u003c/li\u003e\n\u003cli\u003eZhu, D. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv.org. https://arxiv.org/abs/2304.10592(2023)\u003c/li\u003e\n\u003cli\u003eFu, C. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language models. arXiv.org. https://arxiv.org/abs/2306.13394 (2023)\u003c/li\u003e\n\u003cli\u003eHu, Z. LLM-Adapters: an adapter family for Parameter-Efficient Fine-Tuning of large language models. arXiv.org. https://arxiv.org/abs/2304.01933 (2023)\u003c/li\u003e\n\u003cli\u003eLialin, V. Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning. arXiv.org. https://arxiv.org/abs/2303.15647 (2023)\u003c/li\u003e\n\u003cli\u003eLi, X. L. Prefix-Tuning: Optimizing continuous prompts for generation. arXiv.org. https://arxiv.org/abs/2101.00190(2021)\u003c/li\u003e\n\u003cli\u003eHu, E. J. LORA: Low-Rank adaptation of Large Language Models. arXiv.org. https://arxiv.org/abs/2106.09685(2021)\u003c/li\u003e\n\u003cli\u003eWang, Y. PandaLM: an automatic evaluation benchmark for LLM instruction tuning optimization. arXiv.org. https://arxiv.org/abs/2306.05087 (2023)\u003c/li\u003e\n\u003cli\u003eWoo, D. J., Wang, Y., Susanto, H., \u0026amp; Guo, K. Understanding English as a Foreign language Students\u0026rsquo; idea generation Strategies for creative writing with natural language Generation tools. Journal of Educational Computing Research, 073563312311759. https://doi.org/10.1177/07356331231175999Thangaratinam, S., \u0026amp; Redman, C. W. E. (2005). The Delphi technique. The Obstetrician \u0026amp; Gynaecologist, 7(2), 120\u0026ndash;125. https://doi.org/10.1576/toag.7.2.120.27071 (2023)\u003c/li\u003e\n\u003cli\u003eMei, X. WAVCapS: a CHATGPT-Assisted Weakly-Labelled audio captioning dataset for Audio-Language multimodal research. arXiv.org. https://arxiv.org/abs/2303.17395 (2023)\u003c/li\u003e\n\u003cli\u003eLi, J. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv.org. https://arxiv.org/abs/2301.12597 (2023)\u003c/li\u003e\n\u003cli\u003eKhorasani, M., Abdou, M., \u0026amp; Fern\u0026aacute;ndez, J. H. Streamlit use cases. In Apress eBooks (pp. 309\u0026ndash;361). https://doi.org/10.1007/978-1-4842-8111-6_11(2022)\u003c/li\u003e\n\u003cli\u003eYusuf, A., Noor, N. M., \u0026amp; Bello, S. Using multimodal learning analytics to model students\u0026rsquo; learning behavior in animated programming classroom. Education and Information Technologies. https://doi.org/10.1007/s10639-023-12079-8 (2023)\u003c/li\u003e\n\u003cli\u003eLiu, M., Ren, Y., Nyagoga, L. M., Stonier, F., Wu, Z., \u0026amp; Liang, Y. Future of education in the era of generative artificial intelligence: Consensus among Chinese scholars on applications of ChatGPT in schools. Future in Educational Research, 1(1), 72\u0026ndash;101. https://doi.org/10.1002/fer3.10 (2023)\u003c/li\u003e\n\u003cli\u003eLiu, H. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. arXiv.org. https://arxiv.org/abs/2205.05638(2022)\u003c/li\u003e\n\u003cli\u003eZhang, T., Kishore, V., Wu, F., Weinberger, K. Q., \u0026amp; Artzi, Y. BERTScore: Evaluating Text Generation with BERT. arXiv (Cornell University). https://doi.org/10.48550/arxiv.1904.09675 (2019)\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003eTables 1, 2 and 6 are available in the Supplementary Files section.\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Multimodal Large Language Models, Artificial Intelligence, Teacher-Student Interaction, Behavior Analysis","lastPublishedDoi":"10.21203/rs.3.rs-7620633/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7620633/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eTeacher\u0026ndash;student interaction is central to classroom learning, yet traditional observation and machine-learning approaches often remain inefficient and subjective. This study explores the use of multimodal large language models (MLLMs) for systematic analysis of classroom dynamics. We fine-tuned VisualGLM-6B on 2,380 annotated images from 30 classroom videos, covering five interaction types: guided, collaborative, questioning, independent, and exhibitive. LoRA-based fine-tuning combined with prompt engineering was employed to enhance interpretability and domain-specific accuracy. Model performance was assessed through confusion matrices, BERTScore, and expert comparisons. The fine-tuned model achieved 82% overall accuracy, performing best on guided, independent, and exhibitive interactions, while collaborative and questioning types remained challenging. Compared with expert annotation, the model provided more structured and interpretable outputs, though occasional misclassifications and hallucinations persisted. These findings demonstrate the feasibility of applying MLLMs for efficient, objective analysis of teacher\u0026ndash;student interactions and highlight future directions such as incorporating audio inputs and larger datasets to further advance educational research methodologies.\u003c/p\u003e","manuscriptTitle":"Exploring Teacher-Student Interaction through Multimodal Large Language Models: An Empirical Investigation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-10-03 17:12:00","doi":"10.21203/rs.3.rs-7620633/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-01-05T09:23:43+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-05T04:43:20+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"243466083133620702433320382573935169823","date":"2025-10-15T14:33:02+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-13T13:15:19+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"103487223108565809904123518190567826166","date":"2025-10-10T12:07:47+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-09-22T09:44:19+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-09-22T09:43:20+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-09-22T09:22:50+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-09-20T06:48:17+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2025-09-20T06:44:58+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"259881f7-13cd-4520-a4f2-86c40f9c8802","owner":[],"postedDate":"October 3rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":55666914,"name":"Physical sciences/Engineering"},{"id":55666915,"name":"Physical sciences/Mathematics and computing"}],"tags":[],"updatedAt":"2026-02-09T16:05:09+00:00","versionOfRecord":{"articleIdentity":"rs-7620633","link":"https://doi.org/10.1038/s41598-026-38626-0","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2026-02-06 15:59:55","publishedOnDateReadable":"February 6th, 2026"},"versionCreatedAt":"2025-10-03 17:12:00","video":"","vorDoi":"10.1038/s41598-026-38626-0","vorDoiUrl":"https://doi.org/10.1038/s41598-026-38626-0","workflowStages":[]},"version":"v1","identity":"rs-7620633","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7620633","identity":"rs-7620633","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}