MLKD-CLIP：Multi-layer Feature Knowledge Distillation of CLIP for Open-Vocabulary Action Recognition

MLKD-CLIP：Multi-layer Feature Knowledge Distillation of CLIP for Open-Vocabulary Action Recognition | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article MLKD-CLIP：Multi-layer Feature Knowledge Distillation of CLIP for Open-Vocabulary Action Recognition Jingjing Wang, Junyong Ye, Xinyuan Liu, Youwei Li, Guangyi Xu, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5330691/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 16 May, 2025 Read the published version in Multimedia Systems → Version 1 posted 11 You are reading this latest preprint version Abstract Open-vocabulary action recognition aims to identify unseen action categories during training, which is crucial for enabling models to address diverse action scenarios and enhancing their generalizability and adaptability to dynamic environments. Large-scale vision-language pre-trained models (such as CLIP) excel in zero-shot image tasks due to their strong generalizability. However, their lack of temporal information hampers direct application to video tasks. Many studies have fine-tuned CLIP on video datasets, but video datasets are much smaller than the pre-training datasets, potentially leading to reduced generalizability and challenges in recognizing unseen actions. To this end, we propose MLKD-CLIP, which uses the frozen CLIP as the teacher and the fine-tuned CLIP as the student to perform multi-layer feature knowledge distillation. Firstly, we introduce a feature fusion module that employs self-attention to merge features from different layers and incorporates a temporal convolution module, enabling the model to maintain the learning of temporal capabilities during the distillation process. Next, we perform layer-wise fusion to combine the multi-layer features of both the teacher and student models, allowing the model to balance their significance in the distillation process. Finally, we distill the fused features, enabling the student model to learn the multi-level features of the teacher model while considering both global representations and local details. Additionally, the classification tasks on video datasets enhance the student model to learn video features. We evaluated the open-vocabulary action recognition capability of MLKD-CLIP on the UCF101, HMDB51, and SSv2 datasets, achieving the best top-1 accuracy compared to popular methods. Video understanding Action recognition Knowledge distillation Open-vocabulary Multi-layer feature fusion Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 1 Introduction Recently, large pre-trained vision-language models (VLMs), such as CLIP[ 1 ] and ALIGN[ 2 ], have demonstrated remarkable generalizability across many downstream tasks. CLIP was trained on 400 million image-text pairs sourced from the internet, enabling it to effectively integrate semantic information between images and text through contrastive learning techniques. As a result, it has exhibited strong zero-shot recognition capabilities in various domains, such as image segmentation[ 37 , 38 ], image editing[ 39 , 40 ], image classification[ 41 , 42 ] and others. Inspired by these successes, many researchers have attempted to apply CLIP to video action recognition. However, videos differ from images in that they contain rich temporal information. Directly applying the pre-trained CLIP model to videos yields poor results because CLIP was pre-trained on image-text pairs and did not learn temporal information. Research has shown that treating videos as a series of frames and fine-tuning the CLIP visual encoder by adding specialized temporal modeling components[ 21 , 22 , 24 ] can help bridge the semantic gap between images and videos to some extent. However, fine-tuning on video datasets introduces additional optimization parameters, which makes the model more inclined toward the fine-tuned video training set. While this improves performance in closed-set tasks, it inevitably sacrifices some of the generalizability of the pre-trained model, leading to suboptimal performance in open-vocabulary tasks. Therefore, we hope to obtain a model in which CLIP can acquire video-specific knowledge to adapt to the video domain while retaining its generalizability, enabling it to recognize new video categories that were not seen during training. This represents a continual learning process, where the goal is to adapt the pre-trained model to new tasks while minimizing the forgetting of prior knowledge. Previous research FROSTER[ 3 ], used the frozen CLIP model as a teacher to perform residual knowledge distillation, which inspired us. In our approach, the frozen CLIP model serves as the teacher, while the student model learns from the teacher's features and is fine-tuned on video data. During training, the student retains the generalizability of CLIP through distillation loss, while the fine-tuning classification loss on video data helps bridge the gap between video and image. Although FROSTER has proposed balancing the two objectives of knowledge distillation and action recognition through residual methods, the fine-tuned model in the distillation process only learns the teacher's final output features. These final output features are often global abstractions of the input data, which may lead to the loss of some local details. Additionally, the final output may not adequately reflect the feature representations extracted by the teacher model at different stages. To address this problem, we propose MLKD-CLIP, a multi-layer feature distillation approach, the overall structure of which is illustrated in Fig. 1 . This method employs a feature fusion module to layer-wise fuse the intermediate features from both the teacher and student models, ultimately using the fused features for knowledge distillation. The multi-layer feature distillation approach fully leverages the feature extraction capabilities of the teacher model, preserving the generalizability of the CLIP model. At the same time, it incorporates temporal convolution to reduce excessive reliance on the teacher's features, ensuring effectiveness on video data. We used the UCF101[ 4 ], HMDB51[ 5 ], and Something-to-Something V2[ 6 ] datasets to comprehensively evaluate the effectiveness of MLKD-CLIP in the base-to-novel open-vocabulary setting. We compared MLKD-CLIP with other methods for open-vocabulary action recognition, and MLKD-CLIP demonstrated superior zero-shot video action recognition performance. In summary, our work can be summarized as follows: (1) We proposed an open-vocabulary action recognition method based on multi-layer knowledge distillation. By extracting intermediate layer features from both the teacher and student models for distillation, we significantly preserve the generalizability of the CLIP pre-trained model. (2) We introduced a feature fusion module that integrates the extracted multi-layer features, enabling the student model to simultaneously focus on both global features and local details. By employing layer-wise fusion, we balanced the distribution of weights across multiple features, allowing the student model to thoroughly learn from the different layers of features provided by the teacher model. (3) We conducted extensive experiments on the UCF101, HMDB51, and Something-to-Something V2 datasets. In the base-to-novel setting, our method achieved the best performance compared to other approaches. 2 Related Work 2.1 Video Action Recognition Video action recognition aims to classify human actions within videos. With the rapid advancement of deep learning technologies, numerous action classification architectures have been proposed, which can be categorized into three types: 2D Convolutional Neural Networks (2D CNNs)[ 7 – 9 , 43 ], 3D Convolutional Neural Networks (3D CNNs)[ 10 – 12 ], and Transformer-based[ 13 ] methods. After the introduction of the Vision Transformer (ViT)[ 14 ], Transformer-based methods[ 15 – 19 ] have gradually become mainstream. These approaches propose effective methods for spatiotemporal modeling and leverage parameter-efficient fine-tuning techniques to transfer knowledge from the image domain to the video domain. Adapterformer[ 20 ], ST-Adapter[ 21 ], and AIM[ 22 ] propose adding lightweight adapters to networks to fine-tune large pre-trained image models. ActionCLIP[ 23 ] fine-tunes the CLIP model by introducing various prompts to assist the model in learning video features. X-CLIP[ 24 ] introduces cross-frame communication attention to incorporate temporal information into the pre-trained model and proposes video-adaptive prompts that effectively leverage the textual information in category labels. While these methods efficiently transfer pre-trained image models to the video domain, they inevitably reduce the generalizability of the pre-trained models during the fine-tuning process, resulting in poorer performance of the fine-tuned models in open-vocabulary action recognition. 2.2 Open-Vocabulary Action Recognition The open-vocabulary action recognition task challenges models to effectively recognize and classify categories that were not encountered during the training phase, underscoring the importance of its generalizability. To address this, the CLIP model, pre-trained on large-scale visual-text pairs, provides significant support for the field of zero-shot learning due to its outstanding generalizability. By training on rich visual and textual data, the CLIP model can learn more universal feature representations, which are crucial for enhancing model performance in zero-shot learning scenarios. As a result, many researchers have applied CLIP to the field of zero-shot image recognition, however, its application in video action recognition remains limited. ViFi-CLIP[ 25 ] fine-tunes the image and text encoders of CLIP by incorporating learnable prompts into both the visual and textual branches to adapt to downstream tasks. Open-VCLIP[ 26 ] proposes a weight interpolation optimization method, which adds connections between the original CLIP model and the current model to regularize the fine-tuning process of CLIP. Recently, FROSTER[ 3 ] cleverly applied the idea of knowledge distillation to zero-shot action recognition, using the frozen CLIP as the teacher network and the fine-tuned CLIP as the student network. It introduced residual feature distillation to balance the objectives of learning teacher features and classifying video actions. However, it only utilized the global features from the final output of CLIP for distillation, which primarily focuses on global structure and high-level semantic information, neglecting many critical local details. Therefore, we propose enhancing the attention of the student model to local action details by learning the multi-layer features of the teacher model. Additionally, by fusing these multi-layer features, we ensure that the student network captures details while still maintaining attention to global information. 3 Method The knowledge distillation method aims to assist the student in learning through the knowledge of the pre-trained teacher. Previous work has only used the global features from the final output of the model for distillation[ 3 ]. While this approach is straightforward and effective, it neglects the importance of shallow-layer local information. In zero-shot recognition tasks, especially when the distinction between different categories relies on certain detailed features, the local detail information retained in intermediate layer features can be highly beneficial. Therefore, we propose a multi-layer feature fusion distillation method, where the core idea is to fuse the multi-layer features of both the teacher and student models separately and use the fused features for distillation. This approach enables the student model to enhance its focus on local details while simultaneously considering global information through fusion. This section provides a detailed explanation of the proposed method. 3.1 Preliminary 3.1.1 Video Understanding of CLIP We use CLIP[ 1 ] as our base model and treat the video as a series of frames. The CLIP image encoder with the ViT[ 14 ] architecture processes the video input frames, where the input video frames are represented as \(\:{x}_{v}\in\:{\mathbb{R}}^{T\times\:3\times\:H\times\:W}\) , with T being the number of input frames and H×W representing the spatial dimensions of each frame. Ultimately, we use the [CLS] token to represent the global features of the input data. The final output feature representation is as follows: $$\:\begin{array}{c}{v}_{out}={f}_{v}\left({x}_{v}\right),\:{\:\:\:\:\:\:v}_{out}\in\:{\mathbb{R}}^{T\times\:D}\#（1）\end{array}$$ In this representation, D denotes the dimension of the [CLS] token. Finally, we perform average pooling on all \(\:{v}_{out}\) from different frames to obtain the global vector \(\:{\stackrel{-}{v}}_{out}\in\:{\mathbb{R}}^{D}\) . According to Open-VCLIP[ 26 ], the student model learns temporal information by extracting information from the corresponding frame and adjacent frames in the self-attention layers of the image encoder. The self-attention layer is represented as follows: $$\:\begin{array}{c}{x}_{s,t}=Softmax\left(\frac{{q}_{s,t}{\left[{K}_{\left(t-1\right)\sim\left(t+1\right)}\right]}^{T}}{\sqrt{d}}\right)\left[{V}_{\left(t-1\right)\sim\left(t+1\right)}\right]\#（2）\end{array}$$ Here, d represents the dimension of the vectors, and \(\:{q}_{s,t}\) denotes the query vector for the s-th token in the t-th frame, \(\:\left[{K}_{\left(t-1\right)\sim(t+1)}\right]\) and \(\:\left[{V}_{\left(t-1\right)\sim(t+1)}\right]\) refer to the key and value matrices of the frames adjacent to the t-th frame. For the text encoder, we designed a template embedding for each action category (e.g., “a video of []”). We input the category matrix, with the template embedding added, into the text encoder. Similar to FROSTER[ 3 ], we used additional auxiliary descriptions to achieve text augmentation. The text feature representation is as follows: $$\:\begin{array}{c}{t}_{out}={f}_{t}\left({x}_{t}\right),\:{\:\:\:\:\:t}_{out}\in\:{\mathbb{R}}^{N\times\:D}\#（3）\end{array}$$ Here, N represents the number of categories. 3.1.2 Knowledge Distillation Hi et al.[ 27 ] first proposed the concept of knowledge distillation. The core idea is to transfer the knowledge from the pre-trained teacher model to the student model, helping the student model to perform better in feature extraction. To achieve more effective knowledge transfer, various forms of distillation have emerged, such as logits-based alignment[ 28 – 31 ], sample relationship matching[ 32 , 33 ], and feature-based alignment methods[ 34 , 35 ]. Feature-based alignment distillation methods aim to align the feature representations of the student model with those of the teacher model as closely as possible. This approach enables the student to learn more of the raw information from the teacher model. In our approach, we use the frozen pre-trained CLIP model as our teacher and the fine-tuned CLIP model as the student. To enable the student model to comprehensively learn the feature extraction process of teacher, balancing local details and global features, we extract multi-layer features from the teacher model. Through our designed fusion method, these multi-layer features are integrated, allowing the student model to approach the feature representations of the teacher while also taking video information into account. 3.2 MLKD-CLIP In this section, we introduce MLKD-CLIP. The core idea is to distill multi-layer features of the teacher and student models. To help the student model focus on both local details and global information, we propose a multi-layer feature fusion method. By using a specially designed feature fusion module, the multi-layer features from the teacher and student models are fused separately and then distilled. Meanwhile, the student model learns video features through the video classification task. The overall structure is shown in Fig. 1 . 3.2.1 Student Fine-tuning For fine-tuning the student model, according to Equations (1) and (3), the video and text are input into the CLIP encoder separately to obtain the fine-tuned visual features \(\:{v}_{out}^{i}\in\:{\mathbb{R}}^{C}\) and text features \(\:{t}_{out}^{j}\in\:{\mathbb{R}}^{C}\) . Finally, the action category is predicted by calculating the cosine similarity between each video and all category texts: $$\:\begin{array}{c}{y}_{i,j}=\:\frac{⟨{v}_{out}^{i},{t}_{out}^{j}⟩}{‖{v}_{out}^{i}‖‖{t}_{out}^{j}‖}\#（4）\end{array}$$ The video data is classified by calculating the cross-entropy loss between the ground truth \(\:{\widehat{y}}_{i,j}\) and the model prediction \(\:{y}_{i,j}\) : $$\:\begin{array}{c}{L}_{vc}=-\frac{1}{N}\sum\:_{i}^{N}\sum\:_{j}^{K}{\widehat{y}}_{i,j}{log}{y}_{i,j}\#（5）\end{array}$$ Here, K represents the total number of categories. 3.2.2 Multi-layer Feature Fusion Knowledge Distillation Layer-wise feature fusion Since the teacher and student models use the same backbone, their intermediate layer features share the same dimensions, allowing us to directly distill these features without the feature mapping. However, extracting multi-layer features for separate distillation learning can cause the student model to overly focus on the features of teacher, making it difficult to fit the video data during fine-tuning. Additionally, if the multi-layer features are directly used for loss calculation(Fig. 2 a), multiple losses will be generated, and assigning appropriate weights to each loss becomes a challenge. Summing the multi-layer features before loss calculation can address the above issues(Fig. 2 b). However, directly adding the multi-layer features ignores the fact that the importance of features from different layers varies for the distillation task. Therefore, we propose a feature fusion module to layer-wise fuse the extracted multi-layer features. Instead of applying the distillation loss separately at each layer, we conduct unified learning using the features after layer-wise fusion(Fig. 3 ). This process can be expressed as follows: $$\:\begin{array}{c}{f}_{l}^{fusion}={f}_{l}^{student}+{f}_{l-i}^{fusion}\#（6）\end{array}$$ In MLKD-CLIP, we extract and fuse the features from the 2nd, 6th, 8th, and 12th layers of both the teacher and student models. Feature fusion module Common methods for feature fusion typically include weighted fusion and concatenation fusion. Directly applying weighted summation to multi-layer features is a straightforward fusion method. However, selecting appropriate weights for the features of each layer poses a challenge. Concatenation fusion can preserve the original information of the features, but it can incur significant computational overhead. Therefore, we employ the self-attention for fusion. The self-attention can adaptively assign different weights to the features, allowing the model to focus on the key features(Fig. 4 ). Additionally, to further learn temporal information from the videos and ensure the fitting ability of the student model for the video data, we add a temporal modeling module before the self-attention. To save memory, we employ depthwise separable temporal convolutions. This module first uses a 1×1×1 pointwise convolution to change the dimensions of the input features, followed by a 3×1×1 depthwise separable convolution, which convolves along the temporal dimension to capture temporal information and dynamic changes. Finally, a 1×1×1 convolution is used to restore the dimensions. Since the convolution kernel has a spatial dimension of 1×1, it does not alter the spatial features. Meanwhile, each channel undergoes independent convolution operations, preserving the independent features of each channel and significantly reducing both the computational load and parameter count. Loss function The teacher features and student features obtained after the fusion module are represented as and respectively. The distillation loss can be expressed as $$\:\begin{array}{c}{L}_{kD}=\frac{1}{n}\sum\:_{i=1}^{n}{\left({x}_{s}-{x}_{t}\right)}^{2}\#（7）\end{array}$$ By combining the classification loss and the distillation loss, the final loss function can be expressed as: $$\:\begin{array}{c}L={L}_{vc}+\alpha\:{L}_{KD},\#（8）\end{array}$$ where \(\:\alpha\:\) is a balancing coefficient. 4 Experiment 4.1 Datasets UCF101 UCF101[ 4 ] contains 13,320 video clips collected from YouTube, representing 101 action categories. Each video clip in the dataset has a duration of approximately 2 to 10 seconds. The dataset includes 9,537 samples for the training set and 3,783 samples for the testing set, with three different official splits provided. HMDB51 HMDB51[ 5 ] contains 6,489 video clips sourced from movies, YouTube, and other videos, representing 51 action categories. The official dataset also provides three different splits, with each split including 70 training clips and 30 validation clips for all categories. Something-to-Something V2 (SSv2) SSv2[ 6 ] comprises 174 action categories, with 168,913 video clips in the training set and 24,777 video clips in the testing set. This dataset primarily documents interactions between people and objects in everyday life, resulting in more complex scenes and actions that present greater temporal challenges. 4.2 Experimental setting We conducted experiments using the CLIP model based on ViT-B/16. For the teacher model, we utilized the frozen CLIP, while for the student model, we employed Open-VCLIP[ 26 ]. Following ViFi-CLIP[ 25 ] and FROSTER[ 3 ], we report our experimental results under the base-to-novel setting on the SSv2, HMDB51, and UCF101 datasets. In this setting, we divide the categories into two distinct sets: the base set \(\:{Y}_{B}\) and the novel set \(\:{Y}_{N}\) , where the categories do not overlap, i.e., \(\:{Y}_{B}\bigcap\:{Y}_{N}=\varphi\:\) . The model is trained on the base set and evaluated on both the base and novel classes. We report the Top-1 accuracy and harmonic mean for both the base and novel sets across the three datasets. We conducted all experiments on 4 NVIDIA Corporation GA102 [GeForce RTX 3090] GPUs. The initial learning rate was set to 3.33x10 − 6 , with the cosine scheduler for decay, and the training was performed for 12 epochs on each dataset. For the SSv2 dataset, the batch size was set to 16, while for the UCF101 and HMDB51 datasets, the batch size was set to 32. The balance coefficient for the loss function was set to 2. The remaining settings were the same as those in FROSTER[ 3 ]. 4.3 Main Results We compared MLKD-CLIP with other CLIP fine-tuning methods for action recognition under the base-to-novel setting. Table 1 presents a comparison between MLKD-CLIP and adapter-based fine-tuning methods. The results show that our MLKD-CLIP achieves significant improvements on the novel sets of all three video datasets, compared to ST-Adapter[ 21 ], MLKD-CLIP outperforms it by 6.8% on the novel set of UCF101, 9.5% on HMDB51, and 4.2% on SSv2. Table 2 presents a comparison between MLKD-CLIP and prompt-based fine-tuning methods. In comparison to the X-CLIP[ 24 ] model, MLKD-CLIP achieves a 24.7% improvement on the novel set of UCF101, a 12.9% improvement on the novel set of HMDB51, and a 6% improvement on the novel set of SSv2. From the above results, it is evident that our method significantly outperforms both fine-tuning approaches, demonstrating excellent zero-shot recognition capability. It is demonstrated that MLKD-CLIP significantly helps in retaining the generalizability of the pre-trained CLIP model. Meanwhile, the noticeable improvement in accuracy on the base sets of the three datasets indicates that MLKD-CLIP can balance both generalizability and video tasks. In addition, the comparison of fine-tuning methods shows that the adapter-based fine-tuning method performs better overall than the prompt-based fine-tuning method across the three video datasets. This indicates that specialized temporal modeling for video datasets is necessary. Table 1 Base-to-novel setting. We compared MLKD-CLIP with adapter-based models on the video classification task (Top-1 Acc (%)). “HM” denotes the harmonic mean of the accuracy from the base and novel sets. The results for other methods are sourced from FROSTER. The best results are highlighted in bold , while the second-best results are underlined . UCF101 HMDB51 SSv2 Method Base Novel HM Base Novel HM Base Novel HM Frozen CLIP[ 1 ] 78.5 63.6 70.3 53.3 46.8 49.8 4.9 5.3 5.1 AIM[ 22 ] 89.8 76.4 82.6 64.0 51.6 57.1 8.5 7.9 8.2 ST-Adapter[ 21 ] 85.5 76.8 80.9 65.3 48.9 55.9 9.3 8.4 8.8 MLKD-CLIP 96.3 83.6 89.5 74.5 58.4 65.5 18.4 12.6 15.0 Table 2 Base-to-novel setting. We compared MLKD-CLIP with prompt-based models on the video classification task (Top-1 Acc (%)). “HM” denotes the harmonic mean of the accuracy from the base and novel sets. The results for other methods are sourced from FROSTER. The best results are highlighted in bold , while the second-best results are underlined . UCF101 HMDB51 SSv2 Method Base Novel HM Base Novel HM Base Novel HM Frozen CLIP[ 1 ] 78.5 63.6 70.3 53.3 46.8 49.8 4.9 5.3 5.1 VPT[ 36 ] 90.5 40.4 55.8 46.2 16.0 23.8 8.3 5.3 6.4 X-CLIP[ 24 ] 89.9 58.9 71.2 69.4 45.5 55.0 8.5 6.6 7.4 MLKD-CLIP 96.3 83.6 89.5 74.5 58.4 65.5 18.4 12.6 15.0 Table 3 presents a comparison between MLKD-CLIP and the fully fine-tuned CLIP model, demonstrating that our approach outperforms the other methods. These results indicate that our method effectively transfers the generalizability of the pre-trained CLIP model to video data, thereby enhancing the zero-shot video action recognition ability of the fine-tuned model. Compared to FROSTER[ 3 ], which also employs knowledge distillation methods, our approach achieves superior results across all three datasets. This further validates that multi-layer feature distillation is more effective than global feature distillation, allowing the student model to learn the pre-trained knowledge of the frozen CLIP more efficiently and better preserving the generalizability of the pre-trained model. Furthermore, using the frozen CLIP as the teacher and the fine-tuned CLIP as the student in the knowledge distillation approach yields results that are significantly better than those achieved by solely using CLIP as the fine-tuned model. This indicates that the knowledge distillation method aids in maintaining the generalizability of the pre-trained CLIP, providing an effective strategy for open-vocabulary action recognition tasks. Additionally, we observed that the accuracy on the SSv2 dataset is lower compared to other datasets in the open-vocabulary action recognition task. We believe this is because the SSv2 dataset places greater emphasis on the context and details of the actions, resulting in higher demands on the temporal extraction capabilities of the model. Although we incorporated temporal convolution into the fusion module, the CLIP model was pre-trained on image data, which limits its ability to capture subtle variations in motion. Table 3 Base-to-novel setting. We compared MLKD-CLIP with fully-tuned models on the video classification task (Top-1 Acc (%)). “HM” denotes the harmonic mean of the accuracy from the base and novel sets. The results for other methods are sourced from FROSTER. The best results are highlighted in bold , while the second-best results are underlined . UCF101 HMDB51 SSv2 Method Base Novel HM Base Novel HM Base Novel HM Frozen CLIP[ 1 ] 78.5 63.6 70.3 53.3 46.8 49.8 4.9 5.3 5.1 ActionCLIP[ 23 ] 90.1 58.1 70.7 69.1 37.3 48.5 13.3 10.1 11.5 Open-CLIP[ 26 ] 94.8 77.5 85.3 70.3 50.4 58.7 16.0 11.0 13.0 ViFi-CLIP[ 25 ] 92.9 67.7 78.3 73.8 53.3 61.9 16.2 12.1 13.9 FROSTER[ 3 ] 95.3 80.0 87.0 74.1 58.0 65.1 18.3 12.2 14.6 MLKD-CLIP 96.3 83.6 89.5 74.5 58.4 65.5 18.4 12.6 15.0 4.4 Ablation study Ablation Study on Feature Fusion Methods : Table 4 compares three feature fusion approaches: direct distillation of multi-layer features(Fig. 2 a), distillation after summing multi-layer features(Fig. 2 b), and our layer-wise fusion distillation(Fig. 3 ). The experimental results indicate that the method of directly distilling multiple features achieves performance comparable to residual distillation on the UCF101 and SSv2 datasets. Notably, it improves by 2.2% on the novel set of UCF101 compared to residual distillation, demonstrating that the intermediate layer features of the teacher model can effectively impart knowledge to the student model. However, the performance on the HMDB51 dataset is relatively poor. We believe this is due to the model needing to learn multiple loss functions when directly using multi-layer features for distillation. Given that the HMDB51 dataset is smaller, the model struggles to find an appropriate weight balance during training. Summing the multi-layer features alleviates this issue on HMDB51, but the results on SSv2 are lower than those from direct distillation. This may because simply summing features neglects the varying impact of different layer features on the learning capacity of the model. The experimental results of the layer-wise fusion method demonstrate that our approach can address both of the aforementioned issues, validating the effectiveness of our method. Table 4 Comparison of Different Distillation Methods (Top-1 Acc (%)) UCF101 HMDB51 SSV2 Base Novel HM Base Novel HM Base Novel HM Direct Distillation 95.4 82.2 88.3 73.1 56.4 63.7 18.4 12.3 14.7 Summed Distillation 95.9 82.5 88.7 73.1 57.1 64.1 18.1 11.9 14.4 Fusion Distillation 96.3 83.6 89.5 74.5 58.4 65.5 18.4 12.6 15.0 Study on the Position of Temporal Convolution In Table 5 , we conducted an ablation study on the position of the temporal convolution module. Based on the experimental results, we found that our temporal convolution module has the most significant impact on the SSv2 dataset, indicating its usefulness in extracting fine-grained temporal information. Additionally, we examined the effect of the position of the temporal convolution module on model performance by connecting the convolution module in both parallel and sequential configurations with the self-attention module(Fig. 5 ). The experimental results indicate that the performance is better when the modules are connected in series compared to when they are connected in parallel. We believe this is because, when the temporal convolution and self-attention modules are parallel, there is the semantic gap between the temporal information extracted by the temporal convolution module and the spatial information extracted by the self-attention module, making direct fusion challenging. Additionally, the parallel structure prevents the self-attention from effectively utilizing the temporal information. Table 5 Comparison of Results for Temporal Convolution at Different Positions Across Three Datasets (Top-1 Acc (%)). T-Conv denotes temporal convolution UCF101 HMDB51 SSv2 Base Novel HM Base Novel HM Base Novel HM w/o T-Conv 95.2 82.0 88.1 73.1 57.7 64.5 18.2 12.1 14.5 Parallel 95.8 82.3 88.5 73.7 57.9 64.9 18.5 12.5 14.9 Sequential 96.3 83.6 89.5 74.5 58.4 65.5 18.4 12.6 15.0 Effectiveness of Combination with Other Fine-Tuned Models We combined our multi-layer feature distillation method with adapter-based model fine-tuning methods (ST-Adapter[ 21 ], AIM[ 22 ]), as shown in Table 6 . We observed a significant improvement in accuracy on the open-vocabulary action recognition task when multi-layer feature distillation was applied. This demonstrates that our method has a degree of scalability, capable of compensating for the shortcomings of fine-tuned models in zero-shot action recognition and enhancing the feature representation ability of the model. Table 6 Experimental Results of MLKD-CLIP Combined with Two Fine-Tuned Methods (Top-1 Accuracy (%)) UCF101 HMDB51 SSV2 Base Novel HM Base Novel HM Base Novel HM ST-Adapter[ 21 ] 85.5 76.8 80.9 65.3 48.9 55.9 9.3 8.4 8.8 ST-Adapter w/MLKD-CLIP 96.2 81.5 88.2 72.6 55.3 62.8 16.2 12.1 13.8 AIM[ 22 ] 89.8 76.8 80.9 64.0 51.6 57.1 8.5 7.9 8.2 AIM w/MLKD-CLIP 95.6 80.0 87.1 72.7 55.7 63.1 16.0 12.3 13.9 5 Conclusion In this paper, we proposed MLKD-CLIP, a knowledge distillation model designed for open-vocabulary action recognition tasks. MLKD-CLIP extracts multi-layer features from both the student and teacher models, enabling the student to learn the feature extraction process of the teacher, while focusing on both local details and global representations. This approach effectively retains the generalizability of the pre-trained CLIP model. Furthermore, we propose a feature fusion module that separately fuses the multi-layer features of the teacher and student models, using the fused features for distillation learning. This distillation method simplifies loss computation and balances the allocation of weights among multiple features. Extensive experiments conducted in the base-to-novel setting demonstrate the effectiveness of our approach for open-vocabulary action recognition tasks. Open-vocabulary action recognition requires the model to match videos with category texts, making it a multimodal task. Our method explores the unimodal aspect of vision. In the future, we will further investigate the interactions between visual and textual features. Declarations Author Contribution Conceptualization: J.W.; Methodology: J.W. and J.Y.; Formal analysis and investigation: J.W., J.Y. and X.L.; Writing-original draft preparation: J.W., X.L. and Y.L.; Writing-review and editing: J.W., J.Y., X.L., Y.L., G.X. and C.Z.;Resources: J.Y.; Supervision: J.Y..All authors read and approved the final manuscript. Acknowledgement This research was supported by Chongqing Technological Innovation and Application Development Project(No.cstc2021jscx-gksbX0056)，National Key R&D Program ofChina (No. 2020YFC1522900). We appreciate the funding to support the research. References Alec Radford, J.W., Kim, C., Hallacy, A., Ramesh, et al.: Gabriel Go-h, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, (2021) Chao Jia, Y., Yang, Y., Xia, Y.-T., Chen, Z., Parekh, H., Pham, Q., Le, Y.-H., Sung, Z., Li, Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning. In International Conference on Machine Learning, (2021) Huang, X., Zhou, H., Yao, K., Han, K.: Froster: Frozen clip is a strong teacher for open-vocabulary action recognition. In International Conference on Learning Representations, (2024) Khurram Soomro, A.R., Zamir, Shah, M.: UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, (2012) Hildegard Kuehne, H., Jhuang, E., Garrote, T., Poggio, Serre, T.: Hmdb: a large video database for human motion recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, (2011) Raghav Goyal, S.E., Kahou, V., Michalski, J., Materzynska, S., Westphal, H., Kim, V., Haenel, I., Fruend, P., Yianilos, M., Mueller-Freitag, et al.: The something something video database for learning and evaluating visual common sense. In Proceedings of the IEEE/CVF International Conference on Computer Vision, (2017) Limin Wang, Y., Xiong, Z., Wang, Y., Qiao, D., Lin: Xiaoou Tang, and Luc Van Gool. Temporal Segment Networks: Towards Good Practices for Deep Action Recognition. In European Conference on Computer Vision, (2016) Zhaoyang Liu, D., Luo, Y., Wang, L., Wang, Y., Tai, C., Wang, J., Li, F., Huang, Lu, T.: TEINet: Towards an Efficient Architecture for Video Recognition. In Association for the Advancement of Artificial Intelligence, (2020) Ji Lin, C., Gan, S., Han: TSM: Temporal Shift Module for Efficient Video Understanding. In International Conference on Computer Vision, (2019) Carreira, J., Zisserman, A.: Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In Conference on Computer Vision and Pattern Recognition, (2018) Christoph Feichtenhofer, H., Malik, F.J., He, K.: SlowFast Networks for Video Recognition. In Conference on Computer Vision and Pattern Recognition, (2019) Christoph Feichtenhofer: X3D: Expanding Architectures for Efficient Video Recognition. In Conference on Computer Vision and Pattern Recognition, (2020) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Aidan, N., Gomez: Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Conference and Workshop on Neural Information Processing Systems, (2017) Alexey Dosovitskiy, L., Beyer, A., Kolesnikov, D., Weissenborn, X., Zhai, T., Unterthiner, M., Dehghani, M., Minderer, et al.: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, (2021) Anurag Arnab, M., Dehghani, G., Heigold, C.: Sun, Mario Lučić, Cordelia Schmid. ViViT: A Video Vision Transformer. In International Conference on Computer Vision, (2021) Haoqi, F., Xiong, B., Mangalam, K., Li, Y.: Zhicheng Yan, Jitendra Malik, Christoph Feichtenhofer. Multiscale Vision Transformers. In International Conference on Computer Vision, (2021) Li, K., Wang, Y., Gao, P., Song, G., Liu, Y., Li, H.: Yu Qiao. UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning. In International Conference on Learning Representations, (2022) Tong, Z., Song, Y., Wang, J., Wang, L.: VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. In Conference and Workshop on Neural Information Processing Systems, (2022) Li, K., Wang, Y., Li, Y., Wang, Y., He, Y., Wang, L.M.: Yu Qiao. Unmasked Teacher: Towards Training-Efficient Video Foundation Models. In International Conference on Computer Vision, (2023) Chen, S., Ge, C., Tong, Z., Wang, J., Song, Y., Wang, J.: Ping Luo. AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition. In Conference and Workshop on Neural Information Processing Systems, (2022) Junting Pan, Z., Lin, X., Zhu, J., Shao, H., Li: ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning. In Conference and Workshop on Neural Information Processing Systems, (2022) Taojiannan Yang, Y., Zhu, Y., Xie, A., Zhang, C., Chen, M., Li: AIM: Adapting Image Models for Efficient Video Action Recognition. In International Conference on Learning Representations, (2023) Actionclip, M., Wang, J., Xing, Y., Liu: ActionCLIP: A New Paradigm for Video Action Recognition. In IEEE Trans. Neural Networks Learn. Syst., (2023) Bolin Ni, H., Peng, M., Chen, S., Zhang, G., Meng, J., Fu: Shiming Xiang, Haibin Ling. Expanding Language-Image Pretrained Models for General Video Recognition. In European Conference on Computer Vision, (2022) Hanoona Rasheed, M.U., Khattak, M., Maaz, S., Khan: Fahad Shahbaz Khan. Fine-tuned CLIP Models are Efficient Video Learners. In IEEE Conference on Computer Vision and Pattern Recognition, (2023) Zejia Weng, X., Yang, A., Li, Z., Wu, Y.-G., Jiang: Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimization. In International Conference on Machine Learning, (2023) Hinton, G., Vinyals, O., and Jeff Dean:. Distilling the knowledge in a neural network. In ConferenceWorkshop on Neural Information Processing Systems, (2014) Li, Z., Huang, Y., Chen, D., Luo, T.: Ning Cai, and Zhigeng Pan. Online knowledge distillation via multi-branch diversity enhancement. In Asian Conference on Computer Vision, (2020) Li, Z., Li, X., Yang, L., Zhao, B., Song, R., Luo, L., Li, J., Yang, J.: Curriculum temperature for knowledge distillation. In Association for the Advancement of Artificial Intelligence (2023) Zhang, Y., Xiang, T., Timothy, M., Hospedales, Lu, H.: Deep mutual learning. In IEEE Conference on Computer Vision and Pattern Recognition, (2018) Zhao, B., Cui, Q., Song, R.: Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In IEEE Conference on Computer Vision and Pattern Recognition, (2022) Wonpyo Park, D., Kim, Y., Lu, M., Cho: Relational knowledge distillation. In IEEE Conference on Computer Vision and Pattern Recognition, (2019) Chuanguang Yang, Z., An, L.C., Xu, Y.: Mutual contrastive learning for visual representation Learning. In Association for the Advancement of Artificial Intelligence, (2022) Chen, D., Mei, J.-P., Zhang, H., Wang, C.: Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In IEEE Conference on Computer Vision and Pattern Recognition, (2022) Jing Yang, B., Martinez, A., Bulat, G., Tzimiropoulos, et al.: Knowledge distillation via softmax regression representation learning. In International Conference on Learning Representations, (2021) Chen Ju, T., Han, K., Zheng, Y., Zhang, W.X.: Prompting Visual-Language Models for Efficient Video Understanding. In European Conference on Computer Vision, (2022) Zhaoqing Wang, Y., Lu, Q., Li, X., Tao, Y., Guo, M., Gong: Tongliang Liu. CRIS: CLIP-Driven Referring Image Segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, (2022) Golnaz Ghiasi, X., Gu, Y., Cui, T.-Y., Lin: Scaling Open-Vocabulary Image Segmentation with Image-Level Labels. In European Conference on Computer Vision, (2022) Zheng, W., Li, Q., Guo, X., Wan, P., Wang, Z.: Bridging CLIP and StyleGAN through Latent Alignment for Image Editing. arXiv preprint arXiv:2210.04506, 2022. Katherine Crowson, S., Biderman, D., Kornis, D., Stander, E., Hallahan, L., Castricato, E., Raff: VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance. In European Conference on Computer Vision, (2022) Hairui Yang, N., Wang, H., Li, L., Wang, Z., Wang: Application of CLIP for efficient zero-shot learning. Multimedia Syst. 30 , 219 (2024). https://doi.org/10.1007/s00530-024-01414-9 Wang, W., Sun, Q., Zhang, F., Tang, Y., Liu, J.: Xinlong Wang. Diffusion Feedback Helps CLIP See Better. arXiv preprint arXiv:2407.20171, 2024 Aihua Zhou, Y., Ma, W., Ji, M., Zong, P., Yang, M., Wu, M., Liu: Multi-head attention-based two-stream EfficientNet for action recognition. Multimedia Syst. 29 , 487–498 (2023). https://doi.org/10.1007/s00530-022-00961-3 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 16 May, 2025 Read the published version in Multimedia Systems → Version 1 posted Editorial decision: Revision requested 23 Feb, 2025 Reviews received at journal 11 Feb, 2025 Reviews received at journal 09 Feb, 2025 Reviewers agreed at journal 21 Jan, 2025 Reviewers agreed at journal 19 Jan, 2025 Reviewers agreed at journal 16 Dec, 2024 Reviewers agreed at journal 15 Dec, 2024 Reviewers invited by journal 15 Dec, 2024 Editor assigned by journal 09 Dec, 2024 Submission checks completed at journal 26 Oct, 2024 First submitted to journal 25 Oct, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5330691","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":370705256,"identity":"7096e8d7-9d44-4ed4-8620-1795df979f7e","order_by":0,"name":"Jingjing Wang","email":"","orcid":"","institution":"Key Laboratory of Optoelectronic Technology and Systems of the Ministry of Education, Chongqing University","correspondingAuthor":false,"prefix":"","firstName":"Jingjing","middleName":"","lastName":"Wang","suffix":""},{"id":370705257,"identity":"f3a3dfd6-f4e9-4a16-adbc-be452341bf6f","order_by":1,"name":"Junyong Ye","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAvUlEQVRIiWNgGAWjYPACGwjFQ4KWNNK1HCZBi8H5NYYPfu45L893I4Hxwds2BnlzglpuvDE27Hl223DmjQRmw7ltDIY7GwhqOWMmzXDgdoLBjQQ2ad42hgSDA8RpOQfSwv6bOC3ne0BaDoBtYSZKi+QNtmLDngPJhjPPPGyWnHNOwnADIS185w9vfPDjgJ083/Hkgx/elNnIE7RF4UaGAYR1gLEBSEoQUA8E8v3HH0C1EFY8CkbBKBgFIxQAAC38RZcGAf+yAAAAAElFTkSuQmCC","orcid":"","institution":"Key Laboratory of Optoelectronic Technology and Systems of the Ministry of Education, Chongqing University","correspondingAuthor":true,"prefix":"","firstName":"Junyong","middleName":"","lastName":"Ye","suffix":""},{"id":370705258,"identity":"66b788ef-4314-483b-a91a-88ef431205c8","order_by":2,"name":"Xinyuan Liu","email":"","orcid":"","institution":"Key Laboratory of Optoelectronic Technology and Systems of the Ministry of Education, Chongqing University","correspondingAuthor":false,"prefix":"","firstName":"Xinyuan","middleName":"","lastName":"Liu","suffix":""},{"id":370705259,"identity":"46263eb4-13a6-4cd4-a69e-d95df40a08af","order_by":3,"name":"Youwei Li","email":"","orcid":"","institution":"Key Laboratory of Optoelectronic Technology and Systems of the Ministry of Education, Chongqing University","correspondingAuthor":false,"prefix":"","firstName":"Youwei","middleName":"","lastName":"Li","suffix":""},{"id":370705260,"identity":"c198d6a8-dc7b-4d54-a2ce-6f8e1efc973e","order_by":4,"name":"Guangyi Xu","email":"","orcid":"","institution":"Key Laboratory of Optoelectronic Technology and Systems of the Ministry of Education, Chongqing University","correspondingAuthor":false,"prefix":"","firstName":"Guangyi","middleName":"","lastName":"Xu","suffix":""},{"id":370705261,"identity":"65e869fb-e3a2-49a7-91a9-a7d157cd41e5","order_by":5,"name":"Chaoming Zheng","email":"","orcid":"","institution":"Key Laboratory of Optoelectronic Technology and Systems of the Ministry of Education, Chongqing University","correspondingAuthor":false,"prefix":"","firstName":"Chaoming","middleName":"","lastName":"Zheng","suffix":""}],"badges":[],"createdAt":"2024-10-25 08:23:06","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5330691/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5330691/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s00530-025-01836-z","type":"published","date":"2025-05-16T15:57:41+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":69085369,"identity":"6f004299-53f3-420b-9b8e-c2215394bc6c","added_by":"auto","created_at":"2024-11-15 12:39:02","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":205708,"visible":true,"origin":"","legend":"\u003cp\u003eOverview of MLKD-CLIP\u003c/p\u003e","description":"","filename":"Fig.1.tif.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5330691/v1/4501f52f0736330732b9ec5f.jpg"},{"id":69083398,"identity":"98c85bd0-71e8-4343-a2ba-55f853c8019c","added_by":"auto","created_at":"2024-11-15 12:23:02","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":232607,"visible":true,"origin":"","legend":"\u003cp\u003e(a)Direct Distillation (b)Summed Distillation\u003c/p\u003e","description":"","filename":"Fig.2.tif.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5330691/v1/e9a00eed5bcd877ace3e61ed.jpg"},{"id":69084617,"identity":"440bc9c1-6c7e-4919-9346-179badb033c4","added_by":"auto","created_at":"2024-11-15 12:31:02","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":144692,"visible":true,"origin":"","legend":"\u003cp\u003eLayer-wise feature fusion (Ours)\u003c/p\u003e","description":"","filename":"Fig.3.tif.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5330691/v1/effd6c0773333d92706dbd31.jpg"},{"id":69084614,"identity":"8c6113f2-662d-463a-bd0c-13d0384ebccc","added_by":"auto","created_at":"2024-11-15 12:31:02","extension":"jpg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":128732,"visible":true,"origin":"","legend":"\u003cp\u003eFeature fusion module and temporal convolution module\u003c/p\u003e","description":"","filename":"Fig.4.tif.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5330691/v1/8141d596f73116b0242dd72c.jpg"},{"id":69083395,"identity":"c9655277-18d9-4f71-8bfd-b078c5a7b46d","added_by":"auto","created_at":"2024-11-15 12:23:02","extension":"jpg","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":118219,"visible":true,"origin":"","legend":"\u003cp\u003eStudy on the Position of Temporal Convolution. (a)The temporal convolution module and the self-attention module connected in sequential. (b) The temporal convolution module and the self-attention module connected in parallel.\u003c/p\u003e","description":"","filename":"Fig.5.tif.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5330691/v1/3e6c8d93e2c8ca5dd0e2f5c2.jpg"},{"id":83068066,"identity":"d77fc159-4b95-4ece-9460-4d0a3f1101c1","added_by":"auto","created_at":"2025-05-19 16:09:53","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1845152,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5330691/v1/2816d268-2c69-48dc-a5d2-2cbde752c415.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"MLKD-CLIP：Multi-layer Feature Knowledge Distillation of CLIP for Open-Vocabulary Action Recognition","fulltext":[{"header":"1 Introduction","content":"\u003cp\u003eRecently, large pre-trained vision-language models (VLMs), such as CLIP[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e] and ALIGN[\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e], have demonstrated remarkable generalizability across many downstream tasks. CLIP was trained on 400\u0026nbsp;million image-text pairs sourced from the internet, enabling it to effectively integrate semantic information between images and text through contrastive learning techniques. As a result, it has exhibited strong zero-shot recognition capabilities in various domains, such as image segmentation[\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e, \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e], image editing[\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e, \u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e], image classification[\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e, \u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e] and others. Inspired by these successes, many researchers have attempted to apply CLIP to video action recognition. However, videos differ from images in that they contain rich temporal information. Directly applying the pre-trained CLIP model to videos yields poor results because CLIP was pre-trained on image-text pairs and did not learn temporal information. Research has shown that treating videos as a series of frames and fine-tuning the CLIP visual encoder by adding specialized temporal modeling components[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e] can help bridge the semantic gap between images and videos to some extent. However, fine-tuning on video datasets introduces additional optimization parameters, which makes the model more inclined toward the fine-tuned video training set. While this improves performance in closed-set tasks, it inevitably sacrifices some of the generalizability of the pre-trained model, leading to suboptimal performance in open-vocabulary tasks.\u003c/p\u003e \u003cp\u003eTherefore, we hope to obtain a model in which CLIP can acquire video-specific knowledge to adapt to the video domain while retaining its generalizability, enabling it to recognize new video categories that were not seen during training. This represents a continual learning process, where the goal is to adapt the pre-trained model to new tasks while minimizing the forgetting of prior knowledge.\u003c/p\u003e \u003cp\u003ePrevious research FROSTER[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e], used the frozen CLIP model as a teacher to perform residual knowledge distillation, which inspired us. In our approach, the frozen CLIP model serves as the teacher, while the student model learns from the teacher's features and is fine-tuned on video data. During training, the student retains the generalizability of CLIP through distillation loss, while the fine-tuning classification loss on video data helps bridge the gap between video and image. Although FROSTER has proposed balancing the two objectives of knowledge distillation and action recognition through residual methods, the fine-tuned model in the distillation process only learns the teacher's final output features. These final output features are often global abstractions of the input data, which may lead to the loss of some local details. Additionally, the final output may not adequately reflect the feature representations extracted by the teacher model at different stages. To address this problem, we propose MLKD-CLIP, a multi-layer feature distillation approach, the overall structure of which is illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. This method employs a feature fusion module to layer-wise fuse the intermediate features from both the teacher and student models, ultimately using the fused features for knowledge distillation. The multi-layer feature distillation approach fully leverages the feature extraction capabilities of the teacher model, preserving the generalizability of the CLIP model. At the same time, it incorporates temporal convolution to reduce excessive reliance on the teacher's features, ensuring effectiveness on video data.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eWe used the UCF101[\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e], HMDB51[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], and Something-to-Something V2[\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e] datasets to comprehensively evaluate the effectiveness of MLKD-CLIP in the base-to-novel open-vocabulary setting. We compared MLKD-CLIP with other methods for open-vocabulary action recognition, and MLKD-CLIP demonstrated superior zero-shot video action recognition performance.\u003c/p\u003e \u003cp\u003eIn summary, our work can be summarized as follows:\u003c/p\u003e \u003cp\u003e(1) We proposed an open-vocabulary action recognition method based on multi-layer knowledge distillation. By extracting intermediate layer features from both the teacher and student models for distillation, we significantly preserve the generalizability of the CLIP pre-trained model.\u003c/p\u003e \u003cp\u003e(2) We introduced a feature fusion module that integrates the extracted multi-layer features, enabling the student model to simultaneously focus on both global features and local details. By employing layer-wise fusion, we balanced the distribution of weights across multiple features, allowing the student model to thoroughly learn from the different layers of features provided by the teacher model.\u003c/p\u003e \u003cp\u003e(3) We conducted extensive experiments on the UCF101, HMDB51, and Something-to-Something V2 datasets. In the base-to-novel setting, our method achieved the best performance compared to other approaches.\u003c/p\u003e"},{"header":"2 Related Work","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Video Action Recognition\u003c/h2\u003e \u003cp\u003eVideo action recognition aims to classify human actions within videos. With the rapid advancement of deep learning technologies, numerous action classification architectures have been proposed, which can be categorized into three types: 2D Convolutional Neural Networks (2D CNNs)[\u003cspan additionalcitationids=\"CR8\" citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e], 3D Convolutional Neural Networks (3D CNNs)[\u003cspan additionalcitationids=\"CR11\" citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], and Transformer-based[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e] methods. After the introduction of the Vision Transformer (ViT)[\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e], Transformer-based methods[\u003cspan additionalcitationids=\"CR16 CR17 CR18\" citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e] have gradually become mainstream. These approaches propose effective methods for spatiotemporal modeling and leverage parameter-efficient fine-tuning techniques to transfer knowledge from the image domain to the video domain. Adapterformer[\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e], ST-Adapter[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e], and AIM[\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e] propose adding lightweight adapters to networks to fine-tune large pre-trained image models. ActionCLIP[\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e] fine-tunes the CLIP model by introducing various prompts to assist the model in learning video features. X-CLIP[\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e] introduces cross-frame communication attention to incorporate temporal information into the pre-trained model and proposes video-adaptive prompts that effectively leverage the textual information in category labels. While these methods efficiently transfer pre-trained image models to the video domain, they inevitably reduce the generalizability of the pre-trained models during the fine-tuning process, resulting in poorer performance of the fine-tuned models in open-vocabulary action recognition.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Open-Vocabulary Action Recognition\u003c/h2\u003e \u003cp\u003eThe open-vocabulary action recognition task challenges models to effectively recognize and classify categories that were not encountered during the training phase, underscoring the importance of its generalizability. To address this, the CLIP model, pre-trained on large-scale visual-text pairs, provides significant support for the field of zero-shot learning due to its outstanding generalizability. By training on rich visual and textual data, the CLIP model can learn more universal feature representations, which are crucial for enhancing model performance in zero-shot learning scenarios. As a result, many researchers have applied CLIP to the field of zero-shot image recognition, however, its application in video action recognition remains limited. ViFi-CLIP[\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e] fine-tunes the image and text encoders of CLIP by incorporating learnable prompts into both the visual and textual branches to adapt to downstream tasks. Open-VCLIP[\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e] proposes a weight interpolation optimization method, which adds connections between the original CLIP model and the current model to regularize the fine-tuning process of CLIP. Recently, FROSTER[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e] cleverly applied the idea of knowledge distillation to zero-shot action recognition, using the frozen CLIP as the teacher network and the fine-tuned CLIP as the student network. It introduced residual feature distillation to balance the objectives of learning teacher features and classifying video actions. However, it only utilized the global features from the final output of CLIP for distillation, which primarily focuses on global structure and high-level semantic information, neglecting many critical local details. Therefore, we propose enhancing the attention of the student model to local action details by learning the multi-layer features of the teacher model. Additionally, by fusing these multi-layer features, we ensure that the student network captures details while still maintaining attention to global information.\u003c/p\u003e \u003c/div\u003e"},{"header":"3 Method","content":"\u003cp\u003eThe knowledge distillation method aims to assist the student in learning through the knowledge of the pre-trained teacher. Previous work has only used the global features from the final output of the model for distillation[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. While this approach is straightforward and effective, it neglects the importance of shallow-layer local information. In zero-shot recognition tasks, especially when the distinction between different categories relies on certain detailed features, the local detail information retained in intermediate layer features can be highly beneficial. Therefore, we propose a multi-layer feature fusion distillation method, where the core idea is to fuse the multi-layer features of both the teacher and student models separately and use the fused features for distillation. This approach enables the student model to enhance its focus on local details while simultaneously considering global information through fusion. This section provides a detailed explanation of the proposed method.\u003c/p\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Preliminary\u003c/h2\u003e \u003cdiv id=\"Sec7\" class=\"Section3\"\u003e \u003ch2\u003e3.1.1 Video Understanding of CLIP\u003c/h2\u003e \u003cp\u003eWe use CLIP[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e] as our base model and treat the video as a series of frames. The CLIP image encoder with the ViT[\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e] architecture processes the video input frames, where the input video frames are represented as \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{x}_{v}\\in\\:{\\mathbb{R}}^{T\\times\\:3\\times\\:H\\times\\:W}\\)\u003c/span\u003e\u003c/span\u003e, with T being the number of input frames and H\u0026times;W representing the spatial dimensions of each frame. Ultimately, we use the [CLS] token to represent the global features of the input data. The final output feature representation is as follows:\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e\n$$\\:\\begin{array}{c}{v}_{out}={f}_{v}\\left({x}_{v}\\right),\\:{\\:\\:\\:\\:\\:\\:v}_{out}\\in\\:{\\mathbb{R}}^{T\\times\\:D}\\#（1）\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eIn this representation, D denotes the dimension of the [CLS] token. Finally, we perform average pooling on all \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{v}_{out}\\)\u003c/span\u003e\u003c/span\u003e from different frames to obtain the global vector \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\stackrel{-}{v}}_{out}\\in\\:{\\mathbb{R}}^{D}\\)\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eAccording to Open-VCLIP[\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e], the student model learns temporal information by extracting information from the corresponding frame and adjacent frames in the self-attention layers of the image encoder. The self-attention layer is represented as follows:\u003cdiv id=\"Equb\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equb\" name=\"EquationSource\"\u003e\n$$\\:\\begin{array}{c}{x}_{s,t}=Softmax\\left(\\frac{{q}_{s,t}{\\left[{K}_{\\left(t-1\\right)\\sim\\left(t+1\\right)}\\right]}^{T}}{\\sqrt{d}}\\right)\\left[{V}_{\\left(t-1\\right)\\sim\\left(t+1\\right)}\\right]\\#（2）\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eHere, d represents the dimension of the vectors, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{q}_{s,t}\\)\u003c/span\u003e\u003c/span\u003e denotes the query vector for the s-th token in the t-th frame, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\left[{K}_{\\left(t-1\\right)\\sim(t+1)}\\right]\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\left[{V}_{\\left(t-1\\right)\\sim(t+1)}\\right]\\)\u003c/span\u003e\u003c/span\u003e refer to the key and value matrices of the frames adjacent to the t-th frame.\u003c/p\u003e \u003cp\u003eFor the text encoder, we designed a template embedding for each action category (e.g., \u0026ldquo;a video of []\u0026rdquo;). We input the category matrix, with the template embedding added, into the text encoder. Similar to FROSTER[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e], we used additional auxiliary descriptions to achieve text augmentation. The text feature representation is as follows:\u003cdiv id=\"Equc\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equc\" name=\"EquationSource\"\u003e\n$$\\:\\begin{array}{c}{t}_{out}={f}_{t}\\left({x}_{t}\\right),\\:{\\:\\:\\:\\:\\:t}_{out}\\in\\:{\\mathbb{R}}^{N\\times\\:D}\\#（3）\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eHere, N represents the number of categories.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section3\"\u003e \u003ch2\u003e3.1.2 Knowledge Distillation\u003c/h2\u003e \u003cp\u003eHi et al.[\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e] first proposed the concept of knowledge distillation. The core idea is to transfer the knowledge from the pre-trained teacher model to the student model, helping the student model to perform better in feature extraction. To achieve more effective knowledge transfer, various forms of distillation have emerged, such as logits-based alignment[\u003cspan additionalcitationids=\"CR29 CR30\" citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e], sample relationship matching[\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e, \u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e], and feature-based alignment methods[\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e, \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e]. Feature-based alignment distillation methods aim to align the feature representations of the student model with those of the teacher model as closely as possible. This approach enables the student to learn more of the raw information from the teacher model. In our approach, we use the frozen pre-trained CLIP model as our teacher and the fine-tuned CLIP model as the student. To enable the student model to comprehensively learn the feature extraction process of teacher, balancing local details and global features, we extract multi-layer features from the teacher model. Through our designed fusion method, these multi-layer features are integrated, allowing the student model to approach the feature representations of the teacher while also taking video information into account.\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.2 MLKD-CLIP\u003c/h2\u003e \u003cp\u003eIn this section, we introduce MLKD-CLIP. The core idea is to distill multi-layer features of the teacher and student models. To help the student model focus on both local details and global information, we propose a multi-layer feature fusion method. By using a specially designed feature fusion module, the multi-layer features from the teacher and student models are fused separately and then distilled. Meanwhile, the student model learns video features through the video classification task. The overall structure is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cdiv id=\"Sec10\" class=\"Section3\"\u003e \u003ch2\u003e3.2.1 Student Fine-tuning\u003c/h2\u003e \u003cp\u003eFor fine-tuning the student model, according to Equations (1) and (3), the video and text are input into the CLIP encoder separately to obtain the fine-tuned visual features \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{v}_{out}^{i}\\in\\:{\\mathbb{R}}^{C}\\)\u003c/span\u003e\u003c/span\u003e and text features \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{t}_{out}^{j}\\in\\:{\\mathbb{R}}^{C}\\)\u003c/span\u003e\u003c/span\u003e. Finally, the action category is predicted by calculating the cosine similarity between each video and all category texts:\u003cdiv id=\"Equd\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equd\" name=\"EquationSource\"\u003e\n$$\\:\\begin{array}{c}{y}_{i,j}=\\:\\frac{\u0026lang;{v}_{out}^{i},{t}_{out}^{j}\u0026rang;}{‖{v}_{out}^{i}‖‖{t}_{out}^{j}‖}\\#（4）\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eThe video data is classified by calculating the cross-entropy loss between the ground truth \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\widehat{y}}_{i,j}\\)\u003c/span\u003e\u003c/span\u003e and the model prediction \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{y}_{i,j}\\)\u003c/span\u003e\u003c/span\u003e:\u003cdiv id=\"Eque\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Eque\" name=\"EquationSource\"\u003e\n$$\\:\\begin{array}{c}{L}_{vc}=-\\frac{1}{N}\\sum\\:_{i}^{N}\\sum\\:_{j}^{K}{\\widehat{y}}_{i,j}{log}{y}_{i,j}\\#（5）\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eHere, K represents the total number of categories.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section3\"\u003e \u003ch2\u003e3.2.2 Multi-layer Feature Fusion Knowledge Distillation\u003c/h2\u003e \u003cp\u003e \u003cstrong\u003eLayer-wise feature fusion\u003c/strong\u003e \u003cp\u003eSince the teacher and student models use the same backbone, their intermediate layer features share the same dimensions, allowing us to directly distill these features without the feature mapping. However, extracting multi-layer features for separate distillation learning can cause the student model to overly focus on the features of teacher, making it difficult to fit the video data during fine-tuning. Additionally, if the multi-layer features are directly used for loss calculation(Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003ea), multiple losses will be generated, and assigning appropriate weights to each loss becomes a challenge. Summing the multi-layer features before loss calculation can address the above issues(Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eb). However, directly adding the multi-layer features ignores the fact that the importance of features from different layers varies for the distillation task.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eTherefore, we propose a feature fusion module to layer-wise fuse the extracted multi-layer features. Instead of applying the distillation loss separately at each layer, we conduct unified learning using the features after layer-wise fusion(Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). This process can be expressed as follows:\u003cdiv id=\"Equf\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equf\" name=\"EquationSource\"\u003e\n$$\\:\\begin{array}{c}{f}_{l}^{fusion}={f}_{l}^{student}+{f}_{l-i}^{fusion}\\#（6）\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eIn MLKD-CLIP, we extract and fuse the features from the 2nd, 6th, 8th, and 12th layers of both the teacher and student models.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eFeature fusion module\u003c/strong\u003e \u003cp\u003eCommon methods for feature fusion typically include weighted fusion and concatenation fusion. Directly applying weighted summation to multi-layer features is a straightforward fusion method. However, selecting appropriate weights for the features of each layer poses a challenge. Concatenation fusion can preserve the original information of the features, but it can incur significant computational overhead. Therefore, we employ the self-attention for fusion. The self-attention can adaptively assign different weights to the features, allowing the model to focus on the key features(Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003c/p\u003e \u003cp\u003eAdditionally, to further learn temporal information from the videos and ensure the fitting ability of the student model for the video data, we add a temporal modeling module before the self-attention. To save memory, we employ depthwise separable temporal convolutions. This module first uses a 1\u0026times;1\u0026times;1 pointwise convolution to change the dimensions of the input features, followed by a 3\u0026times;1\u0026times;1 depthwise separable convolution, which convolves along the temporal dimension to capture temporal information and dynamic changes. Finally, a 1\u0026times;1\u0026times;1 convolution is used to restore the dimensions. Since the convolution kernel has a spatial dimension of 1\u0026times;1, it does not alter the spatial features. Meanwhile, each channel undergoes independent convolution operations, preserving the independent features of each channel and significantly reducing both the computational load and parameter count.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eLoss function\u003c/strong\u003e \u003cp\u003eThe teacher features and student features obtained after the fusion module are represented as and respectively. The distillation loss can be expressed as\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv id=\"Equg\" class=\"Equation\"\u003e \u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equg\" name=\"EquationSource\"\u003e\n$$\\:\\begin{array}{c}{L}_{kD}=\\frac{1}{n}\\sum\\:_{i=1}^{n}{\\left({x}_{s}-{x}_{t}\\right)}^{2}\\#（7）\\end{array}$$\u003c/div\u003e \u003c/div\u003e \u003c/p\u003e \u003cp\u003eBy combining the classification loss and the distillation loss, the final loss function can be expressed as:\u003cdiv id=\"Equh\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equh\" name=\"EquationSource\"\u003e\n$$\\:\\begin{array}{c}L={L}_{vc}+\\alpha\\:{L}_{KD},\\#（8）\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\alpha\\:\\)\u003c/span\u003e\u003c/span\u003e is a balancing coefficient.\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"4 Experiment","content":"\u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Datasets\u003c/h2\u003e \u003cp\u003e \u003cstrong\u003eUCF101\u003c/strong\u003e \u003cp\u003eUCF101[\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e] contains 13,320 video clips collected from YouTube, representing 101 action categories. Each video clip in the dataset has a duration of approximately 2 to 10 seconds. The dataset includes 9,537 samples for the training set and 3,783 samples for the testing set, with three different official splits provided.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eHMDB51\u003c/strong\u003e \u003cp\u003eHMDB51[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] contains 6,489 video clips sourced from movies, YouTube, and other videos, representing 51 action categories. The official dataset also provides three different splits, with each split including 70 training clips and 30 validation clips for all categories.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eSomething-to-Something V2 (SSv2)\u003c/strong\u003e \u003cp\u003eSSv2[\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e] comprises 174 action categories, with 168,913 video clips in the training set and 24,777 video clips in the testing set. This dataset primarily documents interactions between people and objects in everyday life, resulting in more complex scenes and actions that present greater temporal challenges.\u003c/p\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Experimental setting\u003c/h2\u003e \u003cp\u003eWe conducted experiments using the CLIP model based on ViT-B/16. For the teacher model, we utilized the frozen CLIP, while for the student model, we employed Open-VCLIP[\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. Following ViFi-CLIP[\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e] and FROSTER[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e], we report our experimental results under the base-to-novel setting on the SSv2, HMDB51, and UCF101 datasets. In this setting, we divide the categories into two distinct sets: the base set \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{Y}_{B}\\)\u003c/span\u003e\u003c/span\u003e and the novel set \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{Y}_{N}\\)\u003c/span\u003e\u003c/span\u003e\u003csub\u003e,\u003c/sub\u003e where the categories do not overlap, i.e., \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{Y}_{B}\\bigcap\\:{Y}_{N}=\\varphi\\:\\)\u003c/span\u003e\u003c/span\u003e. The model is trained on the base set and evaluated on both the base and novel classes. We report the Top-1 accuracy and harmonic mean for both the base and novel sets across the three datasets.\u003c/p\u003e \u003cp\u003eWe conducted all experiments on 4 NVIDIA Corporation GA102 [GeForce RTX 3090] GPUs. The initial learning rate was set to 3.33x10\u003csup\u003e\u0026minus;\u0026thinsp;6\u003c/sup\u003e, with the cosine scheduler for decay, and the training was performed for 12 epochs on each dataset. For the SSv2 dataset, the batch size was set to 16, while for the UCF101 and HMDB51 datasets, the batch size was set to 32. The balance coefficient for the loss function was set to 2. The remaining settings were the same as those in FROSTER[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e].\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e4.3 Main Results\u003c/h2\u003e \u003cp\u003eWe compared MLKD-CLIP with other CLIP fine-tuning methods for action recognition under the base-to-novel setting. Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e presents a comparison between MLKD-CLIP and adapter-based fine-tuning methods. The results show that our MLKD-CLIP achieves significant improvements on the novel sets of all three video datasets, compared to ST-Adapter[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e], MLKD-CLIP outperforms it by 6.8% on the novel set of UCF101, 9.5% on HMDB51, and 4.2% on SSv2. Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e presents a comparison between MLKD-CLIP and prompt-based fine-tuning methods. In comparison to the X-CLIP[\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e] model, MLKD-CLIP achieves a 24.7% improvement on the novel set of UCF101, a 12.9% improvement on the novel set of HMDB51, and a 6% improvement on the novel set of SSv2. From the above results, it is evident that our method significantly outperforms both fine-tuning approaches, demonstrating excellent zero-shot recognition capability. It is demonstrated that MLKD-CLIP significantly helps in retaining the generalizability of the pre-trained CLIP model. Meanwhile, the noticeable improvement in accuracy on the base sets of the three datasets indicates that MLKD-CLIP can balance both generalizability and video tasks. In addition, the comparison of fine-tuning methods shows that the adapter-based fine-tuning method performs better overall than the prompt-based fine-tuning method across the three video datasets. This indicates that specialized temporal modeling for video datasets is necessary.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eBase-to-novel setting. We compared MLKD-CLIP with adapter-based models on the video classification task (Top-1 Acc (%)). \u0026ldquo;HM\u0026rdquo; denotes the harmonic mean of the accuracy from the base and novel sets. The results for other methods are sourced from FROSTER. The best results are highlighted in \u003cb\u003ebold\u003c/b\u003e, while the second-best results are \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eunderlined\u003c/span\u003e.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"10\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c4\" namest=\"c2\"\u003e \u003cp\u003eUCF101\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c7\" namest=\"c5\"\u003e \u003cp\u003eHMDB51\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c10\" namest=\"c8\"\u003e \u003cp\u003eSSv2\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMethod\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFrozen CLIP[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e78.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e63.6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e70.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e53.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e46.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e49.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e4.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e5.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e5.1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAIM[\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e89.8\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e76.4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e82.6\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e64.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e51.6\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e57.1\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e8.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e7.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e8.2\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eST-Adapter[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e85.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e76.8\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e80.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e65.3\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e48.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e55.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e9.3\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e8.4\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e8.8\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMLKD-CLIP\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e96.3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e83.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e89.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e74.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e58.4\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e65.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e18.4\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003e12.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e15.0\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eBase-to-novel setting. We compared MLKD-CLIP with prompt-based models on the video classification task (Top-1 Acc (%)). \u0026ldquo;HM\u0026rdquo; denotes the harmonic mean of the accuracy from the base and novel sets. The results for other methods are sourced from FROSTER. The best results are highlighted in \u003cb\u003ebold\u003c/b\u003e, while the second-best results are \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eunderlined\u003c/span\u003e.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"10\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c4\" namest=\"c2\"\u003e \u003cp\u003eUCF101\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c7\" namest=\"c5\"\u003e \u003cp\u003eHMDB51\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c10\" namest=\"c8\"\u003e \u003cp\u003eSSv2\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMethod\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFrozen CLIP[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e78.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e63.6\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e70.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e53.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e46.8\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e49.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e4.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e5.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e5.1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eVPT[\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e90.5\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e40.4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e55.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e46.2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e16.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e23.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e8.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e5.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e6.4\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eX-CLIP[\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e89.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e58.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e71.2\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e69.4\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e45.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e55.0\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e8.5\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e6.6\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e7.4\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMLKD-CLIP\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e96.3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e83.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e89.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e74.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e58.4\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e65.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e18.4\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003e12.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e15.0\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e presents a comparison between MLKD-CLIP and the fully fine-tuned CLIP model, demonstrating that our approach outperforms the other methods. These results indicate that our method effectively transfers the generalizability of the pre-trained CLIP model to video data, thereby enhancing the zero-shot video action recognition ability of the fine-tuned model. Compared to FROSTER[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e], which also employs knowledge distillation methods, our approach achieves superior results across all three datasets. This further validates that multi-layer feature distillation is more effective than global feature distillation, allowing the student model to learn the pre-trained knowledge of the frozen CLIP more efficiently and better preserving the generalizability of the pre-trained model. Furthermore, using the frozen CLIP as the teacher and the fine-tuned CLIP as the student in the knowledge distillation approach yields results that are significantly better than those achieved by solely using CLIP as the fine-tuned model. This indicates that the knowledge distillation method aids in maintaining the generalizability of the pre-trained CLIP, providing an effective strategy for open-vocabulary action recognition tasks.\u003c/p\u003e \u003cp\u003eAdditionally, we observed that the accuracy on the SSv2 dataset is lower compared to other datasets in the open-vocabulary action recognition task. We believe this is because the SSv2 dataset places greater emphasis on the context and details of the actions, resulting in higher demands on the temporal extraction capabilities of the model. Although we incorporated temporal convolution into the fusion module, the CLIP model was pre-trained on image data, which limits its ability to capture subtle variations in motion.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eBase-to-novel setting. We compared MLKD-CLIP with fully-tuned models on the video classification task (Top-1 Acc (%)). \u0026ldquo;HM\u0026rdquo; denotes the harmonic mean of the accuracy from the base and novel sets. The results for other methods are sourced from FROSTER. The best results are highlighted in \u003cb\u003ebold\u003c/b\u003e, while the second-best results are \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eunderlined\u003c/span\u003e.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"10\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c4\" namest=\"c2\"\u003e \u003cp\u003eUCF101\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c7\" namest=\"c5\"\u003e \u003cp\u003eHMDB51\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c10\" namest=\"c8\"\u003e \u003cp\u003eSSv2\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMethod\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFrozen CLIP[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e78.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e63.6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e70.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e53.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e46.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e49.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e4.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e5.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e5.1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eActionCLIP[\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e90.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e58.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e70.7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e69.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e37.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e48.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e13.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e10.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e11.5\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOpen-CLIP[\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e94.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e77.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e85.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e70.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e50.4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e58.7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e16.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e11.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e13.0\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eViFi-CLIP[\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e92.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e67.7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e78.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e73.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e53.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e61.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e16.2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e12.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e13.9\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFROSTER[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e95.3\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e80.0\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e87.0\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e74.1\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e58.0\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e65.1\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e18.3\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e12.2\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e14.6\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMLKD-CLIP\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e96.3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e83.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e89.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e74.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e58.4\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e65.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e18.4\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003e12.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e15.0\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003e4.4 Ablation study\u003c/h2\u003e \u003cp\u003e \u003cem\u003eAblation Study on Feature Fusion Methods\u003c/em\u003e: Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e compares three feature fusion approaches: direct distillation of multi-layer features(Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003ea), distillation after summing multi-layer features(Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eb), and our layer-wise fusion distillation(Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). The experimental results indicate that the method of directly distilling multiple features achieves performance comparable to residual distillation on the UCF101 and SSv2 datasets. Notably, it improves by 2.2% on the novel set of UCF101 compared to residual distillation, demonstrating that the intermediate layer features of the teacher model can effectively impart knowledge to the student model. However, the performance on the HMDB51 dataset is relatively poor. We believe this is due to the model needing to learn multiple loss functions when directly using multi-layer features for distillation. Given that the HMDB51 dataset is smaller, the model struggles to find an appropriate weight balance during training. Summing the multi-layer features alleviates this issue on HMDB51, but the results on SSv2 are lower than those from direct distillation. This may because simply summing features neglects the varying impact of different layer features on the learning capacity of the model. The experimental results of the layer-wise fusion method demonstrate that our approach can address both of the aforementioned issues, validating the effectiveness of our method.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison of Different Distillation Methods (Top-1 Acc (%))\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"10\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c4\" namest=\"c2\"\u003e \u003cp\u003eUCF101\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c7\" namest=\"c5\"\u003e \u003cp\u003eHMDB51\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c10\" namest=\"c8\"\u003e \u003cp\u003eSSV2\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDirect Distillation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e95.4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e82.2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e88.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e73.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e56.4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e63.7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e18.4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e12.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e14.7\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSummed Distillation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e95.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e82.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e88.7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e73.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e57.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e64.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e18.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e11.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e14.4\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFusion Distillation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e96.3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e83.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e89.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e74.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e58.4\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e65.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e18.4\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003e12.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e15.0\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eStudy on the Position of Temporal Convolution\u003c/strong\u003e \u003cp\u003eIn Table\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e, we conducted an ablation study on the position of the temporal convolution module. Based on the experimental results, we found that our temporal convolution module has the most significant impact on the SSv2 dataset, indicating its usefulness in extracting fine-grained temporal information. Additionally, we examined the effect of the position of the temporal convolution module on model performance by connecting the convolution module in both parallel and sequential configurations with the self-attention module(Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e). The experimental results indicate that the performance is better when the modules are connected in series compared to when they are connected in parallel. We believe this is because, when the temporal convolution and self-attention modules are parallel, there is the semantic gap between the temporal information extracted by the temporal convolution module and the spatial information extracted by the self-attention module, making direct fusion challenging. Additionally, the parallel structure prevents the self-attention from effectively utilizing the temporal information.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison of Results for Temporal Convolution at Different Positions Across Three Datasets (Top-1 Acc (%)). T-Conv denotes temporal convolution\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"10\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c4\" namest=\"c2\"\u003e \u003cp\u003eUCF101\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c7\" namest=\"c5\"\u003e \u003cp\u003eHMDB51\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c10\" namest=\"c8\"\u003e \u003cp\u003eSSv2\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ew/o T-Conv\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e95.2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e82.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e88.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e73.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e57.7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e64.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e18.2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e12.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e14.5\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eParallel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e95.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e82.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e88.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e73.7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e57.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e64.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e18.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e12.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e14.9\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSequential\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e96.3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e83.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e89.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e74.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e58.4\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e65.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e18.4\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003e12.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e15.0\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eEffectiveness of Combination with Other Fine-Tuned Models\u003c/strong\u003e \u003cp\u003eWe combined our multi-layer feature distillation method with adapter-based model fine-tuning methods (ST-Adapter[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e], AIM[\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]), as shown in Table\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e6\u003c/span\u003e. We observed a significant improvement in accuracy on the open-vocabulary action recognition task when multi-layer feature distillation was applied. This demonstrates that our method has a degree of scalability, capable of compensating for the shortcomings of fine-tuned models in zero-shot action recognition and enhancing the feature representation ability of the model.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab6\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 6\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eExperimental Results of MLKD-CLIP Combined with Two Fine-Tuned Methods (Top-1 Accuracy (%))\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"10\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c4\" namest=\"c2\"\u003e \u003cp\u003eUCF101\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c7\" namest=\"c5\"\u003e \u003cp\u003eHMDB51\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c10\" namest=\"c8\"\u003e \u003cp\u003eSSV2\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003eNovel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003eHM\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eST-Adapter[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e85.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e76.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e80.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e65.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e48.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e55.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e9.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e8.4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e8.8\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eST-Adapter\u003c/p\u003e \u003cp\u003ew/MLKD-CLIP\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e96.2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e81.5\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e88.2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e72.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e55.3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e62.8\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e16.2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003e12.1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e13.8\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAIM[\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e89.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e76.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e80.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e64.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e51.6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e57.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e8.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e7.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e8.2\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAIM\u003c/p\u003e \u003cp\u003ew/MLKD-CLIP\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e95.6\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e80.0\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e87.1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e72.7\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e55.7\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e63.1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e16.0\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003e12.3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e13.9\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"5 Conclusion","content":"\u003cp\u003eIn this paper, we proposed MLKD-CLIP, a knowledge distillation model designed for open-vocabulary action recognition tasks. MLKD-CLIP extracts multi-layer features from both the student and teacher models, enabling the student to learn the feature extraction process of the teacher, while focusing on both local details and global representations. This approach effectively retains the generalizability of the pre-trained CLIP model. Furthermore, we propose a feature fusion module that separately fuses the multi-layer features of the teacher and student models, using the fused features for distillation learning. This distillation method simplifies loss computation and balances the allocation of weights among multiple features. Extensive experiments conducted in the base-to-novel setting demonstrate the effectiveness of our approach for open-vocabulary action recognition tasks. Open-vocabulary action recognition requires the model to match videos with category texts, making it a multimodal task. Our method explores the unimodal aspect of vision. In the future, we will further investigate the interactions between visual and textual features.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eConceptualization: J.W.; Methodology: J.W. and J.Y.; Formal analysis and investigation: J.W., J.Y. and X.L.; Writing-original draft preparation: J.W., X.L. and Y.L.; Writing-review and editing: J.W., J.Y., X.L., Y.L., G.X. and C.Z.;Resources: J.Y.; Supervision: J.Y..All authors read and approved the final manuscript.\u003c/p\u003e\u003ch2\u003eAcknowledgement\u003c/h2\u003e\u003cp\u003eThis research was supported by Chongqing Technological Innovation and Application Development Project(No.cstc2021jscx-gksbX0056)，National Key R\u0026amp;D Program ofChina (No. 2020YFC1522900). We appreciate the funding to support the research.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eAlec Radford, J.W., Kim, C., Hallacy, A., Ramesh, et al.: Gabriel Go-h, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, (2021)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChao Jia, Y., Yang, Y., Xia, Y.-T., Chen, Z., Parekh, H., Pham, Q., Le, Y.-H., Sung, Z., Li, Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning. In International Conference on Machine Learning, (2021)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang, X., Zhou, H., Yao, K., Han, K.: Froster: Frozen clip is a strong teacher for open-vocabulary action recognition. In International Conference on Learning Representations, (2024)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKhurram Soomro, A.R., Zamir, Shah, M.: UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, (2012)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHildegard Kuehne, H., Jhuang, E., Garrote, T., Poggio, Serre, T.: Hmdb: a large video database for human motion recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, (2011)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRaghav Goyal, S.E., Kahou, V., Michalski, J., Materzynska, S., Westphal, H., Kim, V., Haenel, I., Fruend, P., Yianilos, M., Mueller-Freitag, et al.: The something something video database for learning and evaluating visual common sense. In Proceedings of the IEEE/CVF International Conference on Computer Vision, (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLimin Wang, Y., Xiong, Z., Wang, Y., Qiao, D., Lin: Xiaoou Tang, and Luc Van Gool. Temporal Segment Networks: Towards Good Practices for Deep Action Recognition. In European Conference on Computer Vision, (2016)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhaoyang Liu, D., Luo, Y., Wang, L., Wang, Y., Tai, C., Wang, J., Li, F., Huang, Lu, T.: TEINet: Towards an Efficient Architecture for Video Recognition. In Association for the Advancement of Artificial Intelligence, (2020)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJi Lin, C., Gan, S., Han: TSM: Temporal Shift Module for Efficient Video Understanding. In International Conference on Computer Vision, (2019)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCarreira, J., Zisserman, A.: Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In Conference on Computer Vision and Pattern Recognition, (2018)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChristoph Feichtenhofer, H., Malik, F.J., He, K.: SlowFast Networks for Video Recognition. In Conference on Computer Vision and Pattern Recognition, (2019)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChristoph Feichtenhofer: X3D: Expanding Architectures for Efficient Video Recognition. In Conference on Computer Vision and Pattern Recognition, (2020)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Aidan, N., Gomez: Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Conference and Workshop on Neural Information Processing Systems, (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlexey Dosovitskiy, L., Beyer, A., Kolesnikov, D., Weissenborn, X., Zhai, T., Unterthiner, M., Dehghani, M., Minderer, et al.: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, (2021)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAnurag Arnab, M., Dehghani, G., Heigold, C.: Sun, Mario Lučić, Cordelia Schmid. ViViT: A Video Vision Transformer. In International Conference on Computer Vision, (2021)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHaoqi, F., Xiong, B., Mangalam, K., Li, Y.: Zhicheng Yan, Jitendra Malik, Christoph Feichtenhofer. Multiscale Vision Transformers. In International Conference on Computer Vision, (2021)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi, K., Wang, Y., Gao, P., Song, G., Liu, Y., Li, H.: Yu Qiao. UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning. In International Conference on Learning Representations, (2022)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTong, Z., Song, Y., Wang, J., Wang, L.: VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. In Conference and Workshop on Neural Information Processing Systems, (2022)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi, K., Wang, Y., Li, Y., Wang, Y., He, Y., Wang, L.M.: Yu Qiao. Unmasked Teacher: Towards Training-Efficient Video Foundation Models. In International Conference on Computer Vision, (2023)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen, S., Ge, C., Tong, Z., Wang, J., Song, Y., Wang, J.: Ping Luo. AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition. In Conference and Workshop on Neural Information Processing Systems, (2022)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJunting Pan, Z., Lin, X., Zhu, J., Shao, H., Li: ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning. In Conference and Workshop on Neural Information Processing Systems, (2022)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTaojiannan Yang, Y., Zhu, Y., Xie, A., Zhang, C., Chen, M., Li: AIM: Adapting Image Models for Efficient Video Action Recognition. In International Conference on Learning Representations, (2023)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eActionclip, M., Wang, J., Xing, Y., Liu: ActionCLIP: A New Paradigm for Video Action Recognition. In IEEE Trans. Neural Networks Learn. Syst., (2023)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBolin Ni, H., Peng, M., Chen, S., Zhang, G., Meng, J., Fu: Shiming Xiang, Haibin Ling. Expanding Language-Image Pretrained Models for General Video Recognition. In European Conference on Computer Vision, (2022)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHanoona Rasheed, M.U., Khattak, M., Maaz, S., Khan: Fahad Shahbaz Khan. Fine-tuned CLIP Models are Efficient Video Learners. In IEEE Conference on Computer Vision and Pattern Recognition, (2023)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZejia Weng, X., Yang, A., Li, Z., Wu, Y.-G., Jiang: Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimization. In International Conference on Machine Learning, (2023)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHinton, G., Vinyals, O., and Jeff Dean:. Distilling the knowledge in a neural network. In ConferenceWorkshop on Neural Information Processing Systems, (2014)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi, Z., Huang, Y., Chen, D., Luo, T.: Ning Cai, and Zhigeng Pan. Online knowledge distillation via multi-branch diversity enhancement. In Asian Conference on Computer Vision, (2020)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi, Z., Li, X., Yang, L., Zhao, B., Song, R., Luo, L., Li, J., Yang, J.: Curriculum temperature for knowledge distillation. In Association for the Advancement of Artificial Intelligence (2023)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang, Y., Xiang, T., Timothy, M., Hospedales, Lu, H.: Deep mutual learning. In IEEE Conference on Computer Vision and Pattern Recognition, (2018)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhao, B., Cui, Q., Song, R.: Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In IEEE Conference on Computer Vision and Pattern Recognition, (2022)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWonpyo Park, D., Kim, Y., Lu, M., Cho: Relational knowledge distillation. In IEEE Conference on Computer Vision and Pattern Recognition, (2019)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChuanguang Yang, Z., An, L.C., Xu, Y.: Mutual contrastive learning for visual representation Learning. In Association for the Advancement of Artificial Intelligence, (2022)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen, D., Mei, J.-P., Zhang, H., Wang, C.: Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In IEEE Conference on Computer Vision and Pattern Recognition, (2022)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJing Yang, B., Martinez, A., Bulat, G., Tzimiropoulos, et al.: Knowledge distillation via softmax regression representation learning. In International Conference on Learning Representations, (2021)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen Ju, T., Han, K., Zheng, Y., Zhang, W.X.: Prompting Visual-Language Models for Efficient Video Understanding. In European Conference on Computer Vision, (2022)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhaoqing Wang, Y., Lu, Q., Li, X., Tao, Y., Guo, M., Gong: Tongliang Liu. CRIS: CLIP-Driven Referring Image Segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, (2022)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGolnaz Ghiasi, X., Gu, Y., Cui, T.-Y., Lin: Scaling Open-Vocabulary Image Segmentation with Image-Level Labels. In European Conference on Computer Vision, (2022)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZheng, W., Li, Q., Guo, X., Wan, P., Wang, Z.: Bridging CLIP and StyleGAN through Latent Alignment for Image Editing. arXiv preprint arXiv:2210.04506, 2022.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKatherine Crowson, S., Biderman, D., Kornis, D., Stander, E., Hallahan, L., Castricato, E., Raff: VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance. In European Conference on Computer Vision, (2022)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHairui Yang, N., Wang, H., Li, L., Wang, Z., Wang: Application of CLIP for efficient zero-shot learning. Multimedia Syst. \u003cb\u003e30\u003c/b\u003e, 219 (2024). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s00530-024-01414-9\u003c/span\u003e\u003cspan address=\"10.1007/s00530-024-01414-9\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang, W., Sun, Q., Zhang, F., Tang, Y., Liu, J.: Xinlong Wang. Diffusion Feedback Helps CLIP See Better. arXiv preprint arXiv:2407.20171, 2024\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAihua Zhou, Y., Ma, W., Ji, M., Zong, P., Yang, M., Wu, M., Liu: Multi-head attention-based two-stream EfficientNet for action recognition. Multimedia Syst. \u003cb\u003e29\u003c/b\u003e, 487\u0026ndash;498 (2023). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s00530-022-00961-3\u003c/span\u003e\u003cspan address=\"10.1007/s00530-022-00961-3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"multimedia-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"mmsj","sideBox":"Learn more about [Multimedia Systems](http://link.springer.com/journal/530)","snPcode":"530","submissionUrl":"https://submission.nature.com/new-submission/530/3","title":"Multimedia Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Video understanding, Action recognition, Knowledge distillation, Open-vocabulary, Multi-layer feature fusion","lastPublishedDoi":"10.21203/rs.3.rs-5330691/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5330691/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eOpen-vocabulary action recognition aims to identify unseen action categories during training, which is crucial for enabling models to address diverse action scenarios and enhancing their generalizability and adaptability to dynamic environments. Large-scale vision-language pre-trained models (such as CLIP) excel in zero-shot image tasks due to their strong generalizability. However, their lack of temporal information hampers direct application to video tasks. Many studies have fine-tuned CLIP on video datasets, but video datasets are much smaller than the pre-training datasets, potentially leading to reduced generalizability and challenges in recognizing unseen actions. To this end, we propose MLKD-CLIP, which uses the frozen CLIP as the teacher and the fine-tuned CLIP as the student to perform multi-layer feature knowledge distillation. Firstly, we introduce a feature fusion module that employs self-attention to merge features from different layers and incorporates a temporal convolution module, enabling the model to maintain the learning of temporal capabilities during the distillation process. Next, we perform layer-wise fusion to combine the multi-layer features of both the teacher and student models, allowing the model to balance their significance in the distillation process. Finally, we distill the fused features, enabling the student model to learn the multi-level features of the teacher model while considering both global representations and local details. Additionally, the classification tasks on video datasets enhance the student model to learn video features. We evaluated the open-vocabulary action recognition capability of MLKD-CLIP on the UCF101, HMDB51, and SSv2 datasets, achieving the best top-1 accuracy compared to popular methods.\u003c/p\u003e","manuscriptTitle":"MLKD-CLIP：Multi-layer Feature Knowledge Distillation of CLIP for Open-Vocabulary Action Recognition","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-11-15 12:22:57","doi":"10.21203/rs.3.rs-5330691/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-02-24T01:57:34+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-02-11T07:41:35+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-02-09T06:27:05+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"291394687414631903370563123210919807091","date":"2025-01-22T02:18:37+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"29620362949430319136226540856128323003","date":"2025-01-19T07:30:58+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"78539312880648589573750197233137592596","date":"2024-12-16T05:52:44+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"57098168690808144382985681012036957568","date":"2024-12-15T22:55:51+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-12-15T05:04:46+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-12-10T02:23:22+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-10-26T14:22:22+00:00","index":"","fulltext":""},{"type":"submitted","content":"Multimedia Systems","date":"2024-10-25T08:08:53+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"multimedia-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"mmsj","sideBox":"Learn more about [Multimedia Systems](http://link.springer.com/journal/530)","snPcode":"530","submissionUrl":"https://submission.nature.com/new-submission/530/3","title":"Multimedia Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"cc2d80f3-6649-4a3a-acc1-cca1eece849c","owner":[],"postedDate":"November 15th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-05-19T16:06:35+00:00","versionOfRecord":{"articleIdentity":"rs-5330691","link":"https://doi.org/10.1007/s00530-025-01836-z","journal":{"identity":"multimedia-systems","isVorOnly":false,"title":"Multimedia Systems"},"publishedOn":"2025-05-16 15:57:41","publishedOnDateReadable":"May 16th, 2025"},"versionCreatedAt":"2024-11-15 12:22:57","video":"","vorDoi":"10.1007/s00530-025-01836-z","vorDoiUrl":"https://doi.org/10.1007/s00530-025-01836-z","workflowStages":[]},"version":"v1","identity":"rs-5330691","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5330691","identity":"rs-5330691","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}