Weakly-supervised Temporal Action Localization using Multi-branch Attention Weighting

Abstract Weakly-supervised temporal action localization trains an action localization model using only video-level labels for training. Owing to the lack of frame-level temporal annotations, most existing weakly-supervised temporal action localiza-tion methods are based on multiple instance learning mechanisms to predict the start and end times of all action instances in an untrimmed video and to classify them. Many localization approaches focus only on the most discriminative regions that contribute to the classification task, but ignore a large number of ambiguous background and context snippets in a video. So that results in poor localization performance. We believe that it is helpful to identify such snippets by modeling the background and context. Therefore, we propose a multi-branch attention weighting network (MAW-Net), which introduces an additional non-action class and integrates a multi-branch attention module to generate action and background attention, respectively. In addition, considering the correlation among context, action, and background, we use the difference of action and background attention to construct context attention. Finally, based on these three types of attention values, we obtain three new class activation sequences that distinguish action, background, and context. This enables our proposed model to effectively suppress the activation of context and background snippets and achieve better localization performance. Extensive experiments were performed on the 1 THUMOS-14 and Activitynet1.3 datasets. The experimental results show that our method is superior to other state-of-the-art methods, and its performance is comparable to those of fully-supervised approaches.