Multimodality Fusion based Topic Detection and Evolution Analysis of Web Videos

Multimodality Fusion based Topic Detection and Evolution Analysis of Web Videos | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Multimodality Fusion based Topic Detection and Evolution Analysis of Web Videos Hengmin Zhu, Hongcheng Wei, Jing Wei, Dongyu Ye This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4190241/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Due to the prevalence of video social media and the increase of user generated content, the Internet is filled with a large amount of unstructured data. Videos often contain multimodal data such as title, tags, images and audios. Therefore, fusion of multimodal features is a valid way for video topic detection. The titles and tags of videos are short and sparse, and they are high level semantics, whereas the audio and images of videos are low level semantics. It is not suitable to represent a video by directly fusing these features. To address the issue, an effective multimodal fusion method based on the transformer model is proposed for detecting video topics. First, video data is crawled from Bilibili platform, and the titles, tags and descriptions of videos are processed by deleting invalid symbols and null values. The audios are converted to text and texts are recognized from video covers. Second, the transformer-based model is applied to fuse the three forms of text from different modalities to represent videos with multi-dimensional vectors. Then the HDBSCAN and hierarchical clustering (HC) are compared by Silhouette coefficient when clustering videos for topic detection. In addition, we compare video topic clustering under multimodal and single-modal. Finally, the intensity and content evolution of video topics over time are analyzed in the paper. Experimental results with the real data collected from Bilibili verify the effectiveness of the proposed method for video topic detection and evolution. Multimodality fusion Topic detection Topic evolution Transformer Web videos Bilibili platform Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Introduction With the rapid development of mobile Internet and video websites like YouTube or BiliBili, video platform as a new medium for information dissemination has become a part of our life, which involves a variety of unstructured content, such as text, images and audio. Faced with a large amount of video data, it is difficult for users to quickly access the key content of videos [1]. As a result, it is essential to detect topics from large volume videos and mine their evolutionary relationships, which is of great significance for finding out what users are interested in and keeping users informed of topics on the Internet. However, the text data of videos alone cannot provide a comprehensive overview of video topics, so it is necessary to combine videos’ multimodal data to detect topics. Topic detection and tracking (TDT) has been studied for decades, and previous work has focused on discovering topics and the evolutionary relationships between topics from text data stream [2], which has achieved a lot of success [3-4]. Moreover, as the research progresses, scholars have paid attention to detect topics from multimedia data like images, audios and videos etc. Topic models which are often used for topic detection of text data like Latent Dirichlet Allocation (LDA) [5] have been proved to have significant effects, whereas they require constant tuning of parameters to optimize the results. And these models cannot be directly used for detection of video data. Video topic detection is an arduous task in the public opinion analysis [6], because the multimedia content often contains unstructured data with multiple modalities, which is difficult to process and computationally intensive. How to fully utilize the multimodal information of videos? How to explore an effective and quick method for video topic detection? How to reveal the evolution of the video topics? These are main concerns of discovering topics of videos and revealing topic evolution trends, and need to be furthermore studied. To tackle the above issues, an effective multimodal fusion method based on the transformer model is proposed for video topic detection. We firstly crawl video data from the Bilibili platform which is one of the most popular video websites in China. A variety of features are extracted from the video data, including text data such as titles, tags and descriptions, audio data and video covers. Then, to transform the low-level semantics of audio and video covers into high-level semantics like video text, we respectively convert audios to text by the automatic speech recognition and identify text information contained in video covers by the optical character recognition. Next, multimodal data of videos are fused based on a pre-trained transformer model, which means videos can be represented by these fused data. Finally, we use HDBSCAN to cluster videos to detect topics and compare it with the hierarchical clustering method (HC). The intensity and content evolution of videos’ topics are also analyzed. The main contributions of the paper can be summarized as follows: Firstly, instead of simply concatenating feature vectors from different modalities directly, we convert the audio and video covers to high-level semantic text separately, which allows alignment among multimodal features, and compensation for sparse text such as titles and tags of videos. Secondly, an effective multimodal fusion method is proposed by using the transformer-based model. It takes full advantage of relations between textual, visual and audio data to generate semantic interactions among the features, so videos can be represented more comprehensively. Finally, we analyze the intensity evolution of topics using topic river diagrams and describe the content evolution of topics through the change of topic words sorted by TF-IDF, which can fully reveal the topics’ evolution in different periods. In a word, the proposed method fully utilizes the multimodality to detect videos topics, and reveal the evolutionary trends of video topics, which can be applied for predicting the popularity and trend of a topic in different time periods, and for optimizing video search and content recommendation. The remainder of the paper is organized as follows: In the next section we review related literatures. In Section 3, The research framework and the proposed methods are introduced. Afterward, experiments are performed in Section 4. We conclude with a discussion of results in Section 5. Related work 2.1 Topic detection Most of the existing work on topic recognition has focused on the textual domain, and common methods include topic discovery based on topic framework [7], text-based clustering algorithm [8-9] and LDA [10-12]. LDA which is widely applied in textual data is an effective method for topic detection. In addition, some scholars have also improved the modeling defect of LDA by adding the time factor [13], and considering that LDA cannot be applied to sparse short text modeling, Li et al. [14] put forward the Bi-term Topic Model (BTM) for short-text topic detection. There is a lot of outstanding studies on textual topic detection, whereas it is still difficult to directly detect topics from multimedia data such as videos. Based on the complex network theory and methods, Gargi et al. applied community detection to online video community on YouTube for content discovery [15]; Cao et al. constructed the video social network based on users’ interaction and proposed a community-driven topic detection model for online videos [16]. They detected video topics from the network perspective, but ignored the multimodal features of videos. The tags and keywords of videos are important information describing the content of videos, some scholars attempted to utilize them to detect video topics [17-18]. Liu et al. used the correlation between video response links and keywords to identify topics [19] and Liu et al. detected topics based on video geographic tags for understanding the temporal and spatial distribution of culture, scene and human behavior [20]. From the above studies, it can be found that video tags and keywords can be used to detect topics from videos, whereas the tags or titles of videos are sparse and noisy and cannot fully describe video content. In [21-22], for videos such as lectures or news, semantic information can be extracted through capturing spoken words by automatic speech recognition to enhance video textual features like tags or titles, and the video topics are detected by LDA. Some scholars sliced and annotated videos through the visual features, and proposed various clustering algorithms to mine topics in videos [23-25]. Detecting topics by video keyframes may classify some keyframes of the same topic to different topics. Besides that, training sets need to be constructed in advance, and topics must be predefined, so new topics are difficult to be captured in time. The above studies only rely on a single modality such as text or visual features, and the detected video topics may not be representative. Considering the multimodality and sparsity of videos, the key to video topic detection is to fuse multimodal features. Chen et al. [26] fused the dense tag groups with near duplicate keyframes (NDKs) to detect web video topics and Zhang et al. [27] also fuse NDKs with event keywords, but their methods still rely strongly on video tags. Fu et al. [28] proposed a multigraph fusion method that fuses the video shots by a hypergraph for video summarization. Similarly, Chu et al. [29] proposed a multimodal graph fusion method for cross-media topic detection. Multimodal data fused by graphs may overlook semantic differences between modalities and features have no semantic interaction with each other. The transformer architecture may be an effective tool for fusing multimodal features of video, which has empowered language models to transfer large quantities of data to low-resource domains [30]. And it trains short text data to obtain concise text representation with rich semantics [31]. Rani [32] used the transformer model to fuse heterogeneous data (title, tag and description) for the discovery of web video topics. Zhu [33] adopted a transformer-based encoder-decoder architecture fusing the topical and commonsense features to detect emotion in dialogue text. However, the above studies only fuse textual features, and there are hardly any papers on video topic detection that use transformer-based models to fuse multimodal features. 2.2 Topic evolution Topic evolution is important for capturing trends in related fields [34], and it is often applied to scientific literature or web text. Some studies apply co-word analysis to topic evolution [35-36], which is based on the quantitative calculation of keyword frequency and co-occurrence frequency, and aggregates into topic clusters according to the correlation strength between keyword pairs. It is a common method to investigate topic evolution by measuring the similarity between different topics, and then explore the evolution process between topics [37-38]. In recent years, some scholars believe that topic evolution can be analyzed from multiple dimensions such as structure, intensity and content [39-40]. On this basis, Zhu et al. proposed a ‘word-topic’ coupling network to analyze the process of topic [41] in order to comprehensively describe the evolution process and state of topics. Compared to the field of textual information, there are very few studies on the evolution of video topics. Ryosuke et al. proposed a topic evolution tracking algorithm based on salient keyword matching with consideration of semantic broadness for web videos, and tracked the topic evolution from the hierarchy of video groups [42], which can make users understand the topic evolution over time to find web videos they are interested in, but the calculation cost is high and the amount of data is large. Cao et al. presented an efficient algorithm based on salient trajectory extraction on a topic evolution link graph, which considers tag and visual similarities [43]. There is a lot of work on topic evolution based on text data, and the analytical dimension includes the topic intensity and content, but there is little work on video topic evolution. In summary, there is limited research on topic analysis for multimedia such as videos. Table 1 summarizes the recent studies on video topic detection and evolution. It can be seen from the table that video features for topic detection can be roughly divided into two categories: single-modality and multimodality of videos. The former just considers the text or keyframes of videos, which means that the obtained topics are not well illustrative. The existing work on video multimodal fusion does not consider the semantic level differences among modalities and there is no semantic interaction between features. In this paper, we convert the different features of videos into high-level textual features with rich semantics, and to realize semantic interactions among multimodalities, the transformer-based model is used to fuse features. The HDBSCAN is applied to cluster videos to obtain topics, and the comparative experiments of different clustering methods, single-modal and multimodal are performed respectively. Furthermore, the evolution of video topics is analyzed from the perspective of intensity and content. Table 1. Comparison of previous studies on the video topic detection and evolution. The letter Y and N represent ‘with and without the referred research point’ respectively. References Features Methods of topic detection Content evolution Intensity evolution Liu et al. [17] Keywords Bipartite graph model N N Li et al. [23] Keyframes; Tags Semi-supervised machine learning N N Ryosuke et al. [42] Keywords; Keyframes Keyword matching Network analysis N Cao et al. [43] Tags Event Mining; Detection of burst tags Topic track diagram N Rani et al. [32] Text BERT; HDBSCAN N N Current study Text; Audio; Covers Transformer; HDBSCAN Topic word-based Topic river diagram Methodology In this section, we will introduce the detail of the proposed video topic detection method, which is divided into three phases: data acquisition and video representation, topic detection and evolution analysis. Specifically, data acquisition and video representation involve acquiring video metadata, transforming the different modalities into high-level semantic text, and videos are represented by high-dimensional vectors with 1 row and 768 columns like [0.1444 -0.062 … 0.096 -0.072], which are learned from the fused texts based on the transformer encoder. Topic detection is mainly performed by the HDBSCAN clustering method, and it is compared with hierarchical clustering. We respectively illustrate the evolution of topics in terms of the intensity and content. The intensity is measured by the number of videos and the content evolution is revealed by the change of keywords sorted by TF-IDF. The workflow is shown in Fig. 1 and the proposed method of topic detection is depicted in algorithm 1. 3.1 Preprocessing In order to improve the accuracy and performance of the proposed method, preprocessing is essential for videos’ titles, tags and description texts, audios and covers. Texts like titles, tags and descriptions are processed by removing invalid symbols, anomalies, null values and then they are combined into one text data. Audios are converted into text by automatic speech recognition (ASR), and the converted audio texts are too long and contain invalid data, so the summary extraction is performed for them. Video covers are applied with optical character recognition (OCR) to detect texts. The process in shown in Fig. 2a. 3.2 Data fusion and embedding As shown in Fig. 2a, data fusion aims to integrate multiple modal features of videos to provide a more comprehensive description of videos. Considering the misalignment and semantic level differences among different modalities of videos, the audios and video covers are transformed into the text form with richer semantics. When performing fusion, add separators ‘ESP’ as a distinction between texts of different modalities, and use ‘CLS’ as a beginning symbol, i.e., ‘CLS’ + ‘video title text’ + ‘ESP’ + ‘audio text’ + ‘ESP’ + ‘video cover text’ + ‘ESP’. Video embedding is to convert videos into feature vectors that are learned from the multimodalities. We hope that semantic interactions can be generated among the fused features, resulting in semantically richer fused features. The attention mechanism in the transformer model aims to better understand the contextual relationships of words in texts, which means each word is calculated with other words [32], as shown in Fig. 2b. Therefore, the transformer-based model is used for multimodal fusion to represent videos in the same semantic vector space. Then the similarity between videos can be measured by calculating the cosine similarity of the vectors. 3.3 Topic detection Considering the multimodal features of videos, previous textual topic models cannot be used to detect the video topics, so clustering algorithms are applied for video topic detection in the paper. The clusters are formed by videos with high similarity, and each cluster represents a topic. Since the video vector has high dimensionality, it may not work well in clustering [32]. We firstly reduce the dimensionality and then perform HDBSCAN clustering. The Uniform Manifold Approximation and Projection (UMAP) is employed to achieve it. Compared to some existing dimensionality reduction methods like PCA or t-SNE, UMAP preserves more local and global features of high-dimensional data at a lower projection dimension [44]. And Allaoui et al. [45] demonstrated that the performance of HDBSCAN is improved by reducing high-dimensional embeddings with UMAP, in terms of clustering accuracy and time. Thus, UMAP is used to reduce the dimensionality of embeddings. Introduction of HDBSCAN method. HDBSCAN defines a new distance measurement, the mutual reachability distance that can better reflect the density of points and thus it can handle clustering problems with different densities. When performing clustering, HDBSCAN uses a soft-clustering approach allowing noise to be modeled as outliers, which prevents irrelevant data from being assigned to any cluster and is expected to improve the clustering results [46]. Therefore, HDBSCAN is adopted to cluster similar videos to generate topics. It can be roughly divided into these steps: transforming the space, building the minimum spanning tree, building the cluster hierarchy, condensing the cluster tree and extracting the clusters, as shown in Fig. 3. Introduction of HC method. Hierarchical clustering creates a hierarchical nested clustering tree by calculating the distance between data points of different categories, as shown in Fig. 4a. It does not need to determine the number of clusters in advance, and from the Fig. 4b, it is easy to discover the hierarchical relationship, similar to HDBSCAN. Then the hierarchical clustering method is used for comparison experiments. Using the video similarity as the measure of the distance between videos, the initial video distance matrix is constructed. The distance matrix is updated after each iteration until no new cluster appears, where the distance between different clusters is calculated by average distance. 3.4 Topic evolution The state of topic evolution can be represented by topic intensity and content. Topic intensity can be regarded as the degree of attention and discussion in a period of time [39]. The more the number of blogs to discuss a topic, the higher the hotness of the topic is [38]. For the video topics represented by clusters, the most intuitive way of measuring the topic intensity is the cluster size. When analyzing the evolution of topic content, we can analyze the change of topic words in different time slices. The fused texts of videos in clusters are segmented, and the words are sorted by TF-IDF to get the top ranked keywords. Then the content evolution can be explained based on the change of keywords. Experiments and results 4.1 Experimental dataset and preprocessing We crawl 2067 videos from the issue 147 to issue 192 of the “Weekly Must-see” series on the Bilibili platform, spanning from Jan. 7th, 2022 to Nov. 24th, 2022. In this case, one issue of videos refers to the videos published in one week, so we take two issues (half a month) as a time unit. Each video’s titles, tags and descriptions, cover and audio are crawled and saved. Before topic detection, it is necessary to preprocess the input data. We remove the invalid symbols and null values from the video textual data and combine the processed text. The audios are converted to text by automatic speech recognition and some inaccurate converted texts are corrected manually. The texts in the covers are also recognized by optical character recognition. Then we fuse the data in different modalities and input them into the transformer-based model to get video vectors. 4.2 Performance evaluation In this paper, video topics are detected by clustering methods, which is an unsupervised process. In order to better evaluate and compare the experimental results, Silhouette coefficient (as shown in Equation (1), (2)) is adopted as the experimental evaluation metric, which is a way to evaluate how good the clustering effect is. The larger the Silhouette coefficient, the greater the distance between the cluster structure where the sample point is located and its nearest cluster structure, thus indicating a better clustering result at this point. In order to testify the proposed method, two main comparative experiments are conducted. One is the comparison of clustering methods, and the other is the comparison of different numbers of modalities. The silhouette coefficient of various approaches has been presented in Fig. 5a and Fig. 5b. From the Fig. 5a, it can be concluded that the HDBSCAN is better than the method of HC when the number of clusters is greater than four. And the difference is getting bigger as the number of clusters increases. Furthermore, Fig. 5b shows that the proposed transformer-based multimodal fusion method outperforms single-modal when detecting video topics by HDBSCAN. According to the above comparison results, we can see that HDBSCAN can be used to mine video topics and performs well in the case of multimodal fusion, so it is necessary to make full use of the multimodal features. 4.3 Analysis of topic evolution The state evolution of video topics is analyzed from the perspective of intensity and content respectively. The topics that appear in every time unit (half a month) are selected as the hot topic, and part of hot topics are shown in the Table 2. The topic evolution with a period of three and a half months is analyzed, where the intensity of topics is measured by the number of videos in each cluster and the video content is examined through the keywords sorted by TF-IDF. Table 2. The top 10 keywords of each hot topic in issue 179 to issue 180. The ‘Topic’ column is the id of hot topics. The ‘Video numbers’ column is the number of videos in each cluster. Topic Top 10 keywords Video numbers Topic name T 0 The games; The Black Myth; Game videos; Single-player games; LOL; Players; Sandbox games; Wukong; Live demo; Minecraft, 10 Game T 3 Chinese music; Songs; Originals; MV; Demo; High pitch; Music creation; Music; Classic Songs; Languages 3 Music T 4 Xu Mi; Genshin Impact; Version; Music; Models; Mobile Games; Dubbing; God’s pupil; Scenes; Zhong Li 8 Genshin Impact T 5 Make-up; Ancient style; Red lips; Man in the painting; Pollen; Dream of the red Mansions; Cos; Sexy girl; Immortal Sword; Outfit matching 3 Make-up and clothing T 6 Auto-tune remix-themed content; Cai Xukun; Chicken; Ikun; Planning; Section 3; Fierce man; Cars; Learning to drive; Liu Huaqiang 4 Auto-tune remix-themed content T 7 Food; Midsummer; Tutorial; Cookery; Seafood; Chinese style; Lobsters; Fresh milk; Food ingredients; Cartilaginous fish 10 Food T 11 Animation; Comics; Tom & Jerry; Seminar; Carnival; Styles; Dubbing; Variety comics; Animation News; Painter; 6 Comics and animation T 12 Dancing; Dance moves; Choreography; Divine comedy; Original song; Double dance; Vocaloid; Cosplay; Manpower; Roles 2 Dancing T 13 High energy; Little theatre; Magic; Dark Humor; variety show; Talent show; Brainwashing; Brainstorming; Parody show; Funny 7 Funny videos T 15 Movies; High scores; Emmy Award; American drama; Movie box office; television play; flow rate; Game of Thrones; Film and television; famous scene 2 Film and television T 16 Technology products; Digital; Cell phone; Software; Cooling; Overseas; Domestic mobile phones; App; Records 4 Technology product reviews A topic river diagram is used to visualize the evolution of video topic intensity, as shown in Fig. 6. There are differences in intensity among hot topics in each time unit, like the hot topic of ‘game’ and ‘make-up and clothing’. The intensity of each video topic is not necessarily constant, and it has fluctuations over time. For instance, the topic of ‘Genshin Impact’ has some hotness on September 22, then increases significantly on October 8, but decreases again after November 3. Through the topic river diagram, it is possible to clearly understand the intensity difference among hot topics and examine the intensity change of hot video topics over time. The fused multimodal features are finally presented in the form of text with richer semantics, so in order to reflect the content evolution of the video topics, the text in the clusters can be divided into words, and the words under each topic are sorted by TF-IDF. The evolution of ‘game’ topic from issue 179 to issue192 (three and a half months) is reflected by keywords in the Fig. 7, which shows the top ten keywords ranked by TF-IDF. It can be seen that there exist changes in the content of topic ‘game’ over different periods. Its keywords are not exactly the same in the whole period, which is related to some online hot events. For example, the keywords of ‘LPL’, ‘LOL’ and ‘S12’ are contained in issue 181 to issue 184, because the LOL’s 12th global tournament was held in that period. By examining the changes in keywords, the evolution of video topics can be analyzed. Conclusion Topic detection and tracking in video domain is very interesting. In this paper we propose to use the transformer-based model to fuse multimodal features for detecting video topics. The method fully takes into account the semantic level differences among multimodalities, and enables semantic interaction between features to obtain fused multimodal features with richer semantics. Additionally, the intensity and content evolution of video topics are analyzed based on the river diagram and topic keywords respectively. Experimental results show that the HDBSCAN method is better than the hierarchical clustering in video topic detection. The HDBSCAN just need to set a few parameters while the hierarchical clustering requires setting division thresholds which is susceptible to subjective factors. For video topic detection, the method of fusing multimodal features is better than that of single modality, so it is necessary to make full use of videos’ multimodal features. Analyzing the evolution of video topics in terms of intensity and content allows us to understand the state change of topics on a macro and micro level. The detection and evolution analysis of video topics are conducted through modeling and experiments, and it can keep users informed of online video topics and aid the platform to manage videos with related topics. Furthermore, it is also meaningful for video recommendation, and even for predicting hot topics of videos. Although the current study provides a discussion on hot topic detection and evolution analysis of web videos, it inevitably has its own limitations. The topic detection is based on unsupervised clustering methods, which depends on the setting of parameters, and in the future, self-supervised learning methods that learn the video information from labelled data, can be applied for video topic detection to improve the robustness of the model. In addition, we have fused modalities such as the title, audio and cover of videos, and the video content is also the part of video topics, which may be considered in future research. Declarations Acknowledgements We are very thankful to the editor and people whose valuable suggestions and comments helped us to improve the article. Declaration of conflicting interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Funding This article is supported by the National Natural Science Foundation of China under Grant 72374111, 71874088, and the Postgraduate Research and Practice Innovation Project in Jiangsu Province under the grant KYCX22_0876. Data Availability Statement The data that support the findings of this study are available from the corresponding author, upon reasonable request. References Xie L, Natsev A, Kender J R, et al. Visual memes in social media: tracking real-world news in youtube videos. Proceedings of the 19th ACM international conference on Multimedia. 2011: 53-62. https://doi.org/10.1145/2072298.2072307. Ran L, Suzhi X, Yuanyuan R, et al. A modified approach of hot topics found on micro-blog. Frontier and Future Development of Information Technology in Medicine and Education. Springer, Dordrecht, 2014: 603-614. https://doi.org/10.1007/978-94-007-7618-0_58. Chen K., Luesukprasert L., Chou, S. Hot topic extraction based on timeline analysis and multidimensional sentence modeling, IEEE transactions on knowledge and data engineering. 19 (8) (2007) 1016-1025. 10.3969/j.issn.1005-8095.2020.11.019. Daouadi K., Rebaï R., Amous I. Optimizing semantic deep forest for tweet topic classification, Information Systems. 101 (2001) 101801. https://doi.org/10.1016/j.is.2021.101801. Blei D M, Ng A Y, Jordan M I. Latent dirichlet allocation. Journal of Machine Learning Research, 2003, 3(Jan):993-1022. K.H. Lim, A. Datta. Following the follower: Detecting communities with common interests on Twitter. In: Proceedings of the 23rd ACM conference on Hypertext and social media, Milwaukee, Wisconsin, USA, 25-28 June 2012, pp. 317-318. https://doi.org/10.1145/2309996.2310052. Xie W, Zhu F, Jiang J et al. Topic sketch: Real-time bursty topic detection from twitter, IEEE Transactions on Knowledge and Data Engineering. 28 (8) (2016) 2216-2229. 10.1109/TKDE.2016.2556661. Lu Y, Zhang P, Liu J et al. Health-related hot topic detection in online communities using text clustering, Plos one. 8 (2) (2013) e56221. https://doi.org/10.1371/journal.pone.0056221. Pons A, Berlanga R, Ruiz J. Topic discovery based on text mining techniques, Information processing & management. 43 (3) (2007) 752-768. https://doi.org/10.1016/j.ipm.2006.06.001. Zhao F, Zhu Y, Jin effeeffectH et al. A personalized hashtag recommendation approach using LDA-based topic model in microblog environment, Future Generation Computer Systems. 65 (2016) 196-206. https://doi.org/10.1016/j.future.2015.10.012. Guo X, Xiang Y, Chen Q et al. LDA-based online topic detection using tensor factorization, Journal of information science. 39 (4) (2013) 459-469. https://doi.org/10.1177/0165551512473066. AlSumait L, Barbará D, Domeniconi C. On-line lda: Adaptive topic models for mining text streams with applications to topic detection and tracking. In: 2008 eighth IEEE international conference on data mining, Pisa, Italy,15-19 December 2008, pp. 3-12. https://doi.org/10.1109/ICDM.2008.140. Chen L C. An effective LDA-based time topic model to improve blog search performance. Information Processing & Management, 2017, 53(6): 1299-1319. https://doi.org/10.1016/j.ipm.2017.08.001. Li W, Feng Y, Li D, et al. Micro-blog topic detection method based on BTM topic model and K-means clustering algorithm. Automatic Control and Computer Sciences, 2016, 50(4): 271-277. https://doi.org/10.3103/S0146411616040040. Gargi U, Lu W, Mirrokni V et al. Large-scale community detection on youtube for topic discovery and exploration. In: Fifth International AAAI Conference on Weblogs and Social Media, Barcelona, Catalonia, Spain, 17-21 July 2011, pp. 486-489. http://www.aaai.org/ocs/index.php/ICWSM/ICWSM11/paper/view/2864. Cao J, Zhang Y, Ji R et al. Web video topics discovery and structuralization with social network, Neurocomputing. 172 (2016) 53-63. https://doi.org/10.1016/j.neucom.2014.10.103. Liu L, Sun L, Rui Y et al. Web video topic discovery and tracking via bipartite graph reinforcement model. In: Proceedings of the 17th international conference on World Wide Web, Beijing, China, April 2008, pp. 1009-1018. https://doi.org/10.1145/1367497.1367633. Wang Y, Wu T, Li G et al. Video topic detection on Micro-Blog using Relational Topic Model, Academic Journal of Computing & Information Science. 4 (2) (2021). https://doi.org/10.25236/AJCIS.2021.040214. Liu Y, Yu N. Dual linkage refinement for YouTube video topic discovery. In: 2010 IEEE International Conference on Multimedia and Expo, Singapore, 19-23 July 2010, pp. 1576-1581. 10.1109/ICME.2010.5582943. Liu K, Xu J, Zhang L et al. Discovering hot topics from geo-tagged video, Neurocomputing. 105(2013) 90-99. https://doi.org/10.1016/j.neucom.2012.05.035. Bian J, Huang M L. Semantic topic discovery for lecture video. In: Proceedings of SAI Intelligent Systems Conference, London, UK, September 2019, pp.457-466. https://doi.org/10.1007/978-3-030-29516-5_36. Zuo L. Research on the automatic generation scheme of interview short video titles, Computer Products and Circulation. 11(2017) 158. CNKI:SUN:WXXJ.0.2017-11-144. Li G, Jiang S, Zhang W et al. Online web video topic detection and tracking with semi-supervised learning, Multimedia Systems. 22 (1) (2016) 115-125. https://doi.org/10.1007/s00530-014-0402-0. Shao J, Ma S, Lu W et al. A unified framework for web video topic discovery and visualization, Pattern Recognition Letters. 33 (4) (2012) 410-419. https://doi.org/10.1016/j.patrec.2011.07.026. Shao J, Yin W, Ma S et al. Topic discovery of web video using star-structured k-partite graph. In: Proceedings of the 18th ACM international conference on Multimedia, Firenze, Italy, October 2010, pp. 915-918. https://doi.org/10.1145/1873951.1874112. Chen T, Liu C, Huang Q. An effective multi-clue fusion approach for web video topic detection. Proceedings of the 20th ACM international conference on Multimedia. 2012: 781-784. https://doi.org/10.1145/2393347.2396311. Zhang C, Wu X, Shyu M L, et al. Adaptive association rule mining for web video event classification. 2013 IEEE 14th International Conference on Information Reuse & Integration (IRI). IEEE, 2013: 618-625. https://doi.org/10.1109/IRI.2013.6642526. Fu Y, Guo Y, Zhu Y, et al. Multi-view video summarization. IEEE Transactions on Multimedia, 2010, 12(7): 717-729. https://doi.org/10.1109/TMM.2010.2052025. Chu L, Zhang Y, Li G, et al. Effective multimodality fusion framework for cross-media topic detection. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 26(3): 556-569. https://doi.org/10.1109/TCSVT.2014.2347551. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017(30), pages 5998–6008. Cheng P, Du J, Kou F, et al. Topic Detection Based on Semantics, Time and Social Relationship. Proceedings of 2019 Chinese Intelligent Automation Conference. Springer Singapore, 2020: 691-698. https://doi.org/10.1007/978-981-32-9050-1_78. Rani S, Kumar M. Heterogeneous Information Fusion based Topic Detection from Social Media Data. Information Systems Frontiers, 2022: 1-16. https://doi.org/10.1007/s10796-022-10334-w. Zhu L, Pergola G, Gui L, et al. Topic-driven and knowledge-aware transformer for dialogue emotion detection. arXiv preprint arXiv:2106.01071, 2021. https://doi.org/10.48550/arXiv.2106.01071. Chen B, Tsutsui S, Ding Y et al. Understanding the topic evolution in a scientific domain: An exploratory study for the field of information retrieval, Journal of Informetrics. 11 (4) (2017) 1175-1189. https://doi.org/10.1016/j.joi.2017.10.003. Tang G, Zhang W. Research progress and analysis of subject evolution based on co-word analysis method, Library and Information Service. 59 (5) (2015) 128-136. 10.13266/j.issn.0252-3116.2015.05.020. Wei L, Jiamin W, Jiming H. Analyzing the topic distribution and evolution of foreign relations from parliamentary debates: A framework and case study, Information Processing & Management. 57 (3) (2020) 102191. https://doi.org/10.1016/j.ipm.2019.102191. Lv N, Luo J, Liu Y et al. Analysis of topic evolution based on subtopic similarity. In: 2009 International Conference on Computational Intelligence and Natural Computing, Wuhan, China, 6-7 June 2009, pp. 506-509. 10.1109/CINC.2009.23. Jian F, Ya W, Yuan D. Microblog topic evolution computing based on LDA algorithm, Open Physics. 16 (1) (2018) 509-516. https://doi.org/10.1515/phys-2018-0067. Liu Z, Wang X, Bai R. Construction and Empirical Research on Multi-dimensional Topic Evolution Analysis Model, Information Theory and Practice. 40 (3) (2017) 92-98. 10.16353/j.cnki.1000-7490.2017.03.018. Wang X, Cheng Q, Lu W. Analyzing evolution of research topics with NEViewer: a new method based on dynamic co-word networks, Scientometrics. 101 (2) (2014) 1253-1271. https://doi.org/10.1007/s11192-014-1347-y. Zhu, H., Qian, L., Qin, W. et al. Evolution analysis of online topics based on ‘word-topic’ coupling network. Scientometrics 127, 3767–3792 (2022). https://doi.org/10.1007/s11192-022-04439-x Harakawa R, Ogawa T, Haseyama M. Tracking topic evolution via salient keyword matching with consideration of semantic broadness for Web video discovery, Multimedia Tools and Applications. 77 (16) (2018) 20297-20324. https://doi.org/10.1007/s11042-017-5404-4. Cao J, Ngo C, Zhang Y et al. Tracking web video topics: Discovery, visualization, and monitoring, IEEE Transactions on Circuits and Systems for Video. Technology. 21 (12) (2011) 1835-1846. 10.1109/TCSVT.2011.2148470. McInnes, L., Healy, J., & Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv:1802.03426, 2018. https://doi.org/10.48550/arXiv.1802.03426. Allaoui M, Kherfi M L, Cheriet A. Considerably improving clustering algorithms using UMAP dimensionality reduction technique: a comparative study. Image and Signal Processing: 9th International Conference, ICISP 2020, Marrakesh, Morocco, June 4–6, 2020, 2020: 317-325. https://doi.org/10.1007/978-3-030-51935-3_34. Grootendorst M. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794, 2022. https://doi.org/10.48550/arXiv.2203.05794. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4190241","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":285809904,"identity":"0b698def-ca7f-4bed-85a0-b7f0dba99e06","order_by":0,"name":"Hengmin Zhu","email":"","orcid":"","institution":"Nanjing University of Posts and Telecommunications","correspondingAuthor":false,"prefix":"","firstName":"Hengmin","middleName":"","lastName":"Zhu","suffix":""},{"id":285809905,"identity":"44b644c8-1b68-4acc-824a-912d4d297517","order_by":1,"name":"Hongcheng Wei","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAtklEQVRIiWNgGAWjYDACCQaGw38qbOTY2NsPEK2F8QDPmTRjPp4zCURrYT7A23Y4cZ6EgwFxOgxuNz84INl2OL1NgiGB4UfFNiK03DlmcMDgXHpum3TjAcaeM7cJazG7kcNwIKHMOrdN5kACM2MbsVoOsDGns0kkGBCv5WBDm3MC8Vrsb6QZHGY4k2bYBgzkg0T5RXJG8uPPDBU28vLt7Qcf/KggQgsKOECi+lEwCkbBKBgFuAAAOFJAeTEn5QAAAAAASUVORK5CYII=","orcid":"","institution":"Nanjing University of Posts and Telecommunications","correspondingAuthor":true,"prefix":"","firstName":"Hongcheng","middleName":"","lastName":"Wei","suffix":""},{"id":285809906,"identity":"4319f78d-2b3c-40d4-8d70-2c8dc34aac4c","order_by":2,"name":"Jing Wei","email":"","orcid":"","institution":"Nanjing University of Posts and Telecommunications","correspondingAuthor":false,"prefix":"","firstName":"Jing","middleName":"","lastName":"Wei","suffix":""},{"id":285809907,"identity":"6fc663e9-0833-48db-b8da-c51421b0ad95","order_by":3,"name":"Dongyu Ye","email":"","orcid":"","institution":"Nanjing University of Posts and Telecommunications","correspondingAuthor":false,"prefix":"","firstName":"Dongyu","middleName":"","lastName":"Ye","suffix":""}],"badges":[],"createdAt":"2024-03-30 02:29:24","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4190241/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4190241/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":54115497,"identity":"af6e81fa-1a40-4fcb-a8e2-b2d3c73079ff","added_by":"auto","created_at":"2024-04-04 19:31:49","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":263504,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-4190241/v1/5146b78566a0c7eba34d337c.png"},{"id":54115498,"identity":"6feeaeb2-349b-45d6-90e4-5e437760b3ec","added_by":"auto","created_at":"2024-04-04 19:31:49","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":202603,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-4190241/v1/cbdb3e93ed3929c6a3834091.png"},{"id":54115502,"identity":"1e1e08a1-f765-4410-9959-b86689c22d6e","added_by":"auto","created_at":"2024-04-04 19:31:50","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":271851,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-4190241/v1/72dee1083518490886c5c7ff.png"},{"id":54115501,"identity":"5a90ed60-4c7a-40f9-aff6-e9124bac9ac0","added_by":"auto","created_at":"2024-04-04 19:31:49","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":123914,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-4190241/v1/8cbe3e17d9066fb3267bd317.png"},{"id":54116028,"identity":"100bfeb4-50e4-4109-ab45-f7519bc09fd2","added_by":"auto","created_at":"2024-04-04 19:39:49","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":143098,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-4190241/v1/cbc07aa0d7595253acd3f176.png"},{"id":54115500,"identity":"366014da-7dd2-4a58-9165-6320593c35be","added_by":"auto","created_at":"2024-04-04 19:31:49","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":156223,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-4190241/v1/91e0c9fc44728a957c6be48d.png"},{"id":54115503,"identity":"6c5d49b9-a18a-47e9-acd5-1586d3c79e4a","added_by":"auto","created_at":"2024-04-04 19:31:50","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":22816,"visible":true,"origin":"","legend":"\u003cp\u003eThe keywords changing of the hot topic ‘game’ from issue 179 (Aug. 19, 2022) to issue 192 (Nov. 24, 2022). The divided words of the topic are sorted by TF-IDF and the top ten of them are selected as topic words.\u003c/p\u003e","description":"","filename":"7.png","url":"https://assets-eu.researchsquare.com/files/rs-4190241/v1/61118d55f42efc9062bb40a0.png"},{"id":54787361,"identity":"dd6f1e71-d9df-4b87-a6fb-b31bad9faedb","added_by":"auto","created_at":"2024-04-16 19:06:52","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1505765,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4190241/v1/dcf0f329-d344-4280-a138-367055d4b534.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Multimodality Fusion based Topic Detection and Evolution Analysis of Web Videos","fulltext":[{"header":"Introduction","content":"\u003cp\u003eWith the rapid development of mobile Internet and video websites like YouTube or BiliBili, video platform as a new medium for information dissemination has become a part of our life, which involves a variety of\u0026nbsp;unstructured\u0026nbsp;content, such as text, images and audio. Faced with a large amount of video data, it is difficult for users to quickly access the key content of videos [1]. As a result, it is essential to detect topics from large volume videos and mine their evolutionary relationships, which is of great significance for finding out what users are interested in and keeping users informed of topics on the Internet. However, the text data of videos alone cannot provide a comprehensive overview of video topics, so it is necessary to combine videos\u0026rsquo; multimodal data to detect topics.\u003c/p\u003e\n\u003cp\u003eTopic detection and tracking (TDT) has been studied for decades, and previous work has focused on discovering topics and the evolutionary relationships between topics from text data stream [2], which has achieved a lot of success [3-4]. Moreover, as the research progresses, scholars have paid attention to detect topics from multimedia data like images, audios and videos etc. Topic models which are often used for topic detection of text data like Latent Dirichlet Allocation (LDA) [5] have been proved to have significant effects, whereas they require constant tuning of parameters to optimize the results. And these models cannot be directly used for detection of video data. Video topic detection is an arduous task in the public opinion analysis [6], because the multimedia content often contains unstructured data with multiple modalities, which is difficult to process and computationally intensive. How to fully utilize the multimodal information of videos?\u0026nbsp;How to explore an effective and quick method for video topic detection? How to reveal the evolution of the video topics? These are main concerns of discovering topics of videos and revealing topic evolution trends, and need to be furthermore studied.\u003c/p\u003e\n\u003cp\u003eTo tackle the above issues,\u0026nbsp;an effective multimodal fusion method based on the transformer model is proposed for video topic detection. We firstly crawl video data from the Bilibili platform which is one of the most popular video websites in China. A variety of features are extracted from the video data, including text data such as titles, tags and descriptions, audio data and video covers. Then, to transform the low-level semantics of audio and video covers into high-level semantics like video text, we respectively convert audios to text by the automatic speech recognition and identify text information contained in video covers by the optical character recognition. Next, multimodal data of videos are fused based on a pre-trained transformer model, which means videos can be represented by these fused data. Finally, we use HDBSCAN to cluster videos to detect topics and compare it with the hierarchical clustering method (HC). The intensity and content evolution of videos\u0026rsquo; topics are also analyzed.\u003c/p\u003e\n\u003cp\u003eThe main contributions of the paper can be summarized as follows: Firstly, instead of simply concatenating feature vectors from different modalities directly, we convert the audio and video covers to high-level semantic text separately, which allows alignment among multimodal features, and compensation for sparse text such as titles and tags of videos. Secondly, an effective multimodal fusion method is proposed by using the transformer-based model. It takes full advantage of relations between textual, visual and audio data to generate semantic interactions among the features, so videos can be represented more comprehensively. Finally, we analyze the intensity evolution of topics using topic river diagrams and describe the content evolution of topics through the change of topic words sorted by TF-IDF, which can fully reveal the topics\u0026rsquo; evolution in different periods. In a word, the proposed method fully utilizes the multimodality to detect videos topics, and reveal the evolutionary trends of video topics, which can be applied for predicting the popularity and trend of a topic in different time periods, and for optimizing video search and content recommendation.\u003c/p\u003e\n\u003cp\u003eThe remainder of the paper is organized as follows: In the next section we review related literatures. In Section 3, The research framework and the proposed methods are introduced. Afterward, experiments are performed in Section 4. We conclude with a discussion of results in Section 5.\u003c/p\u003e"},{"header":"Related work","content":"\u003cp\u003e\u003cstrong\u003e2.1 Topic detection\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eMost of the existing work on topic recognition has focused on the textual domain, and common methods include topic discovery based on topic framework [7], text-based clustering algorithm [8-9] and LDA [10-12]. LDA which is widely applied in textual data is an effective method for topic detection. In addition, some scholars have also improved the modeling defect of LDA by adding the time factor [13], and considering that LDA cannot be applied to sparse short text modeling, Li et al. [14] put forward the Bi-term Topic Model (BTM) for short-text topic detection. There is a lot of outstanding studies on textual topic detection, whereas it is still difficult to directly detect topics from multimedia data such as videos.\u003c/p\u003e\n\u003cp\u003eBased on the complex network theory and methods, Gargi et al. applied community detection to online video community on YouTube for content discovery\u003csup\u003e\u0026nbsp;\u003c/sup\u003e[15]; Cao et al. constructed the video social network based on users\u0026rsquo; interaction and proposed a community-driven topic detection model for online videos [16]. They detected video topics from the network perspective, but ignored the multimodal features of videos. The tags and keywords of videos are important information describing the content of videos, some scholars attempted to utilize them to detect video topics [17-18]. Liu et al. used the correlation between video response links and keywords to identify topics [19] and Liu et al. detected topics based on video geographic tags for understanding the temporal and spatial distribution of culture, scene and human behavior [20]. From the above studies, it can be found that video tags and keywords can be used to detect topics from videos, whereas the tags or titles of videos are sparse and noisy and cannot fully describe video content. In [21-22], for videos such as lectures or news, semantic information can be extracted through capturing spoken words by automatic speech recognition to enhance video textual features like tags or titles, and the video topics are detected by LDA. Some scholars sliced and annotated videos through the visual features, and proposed various clustering algorithms to mine topics in videos [23-25]. Detecting topics by video keyframes may classify some keyframes of the same topic to different topics. Besides that, training sets need to be constructed in advance, and topics must be predefined, so new topics are difficult to be captured in time. The above studies only rely on a single modality such as text or visual features, and the detected video topics may not be representative.\u003c/p\u003e\n\u003cp\u003eConsidering the multimodality and sparsity of videos, the key to video topic detection is to fuse multimodal features. Chen et al. [26] fused the dense tag groups with near duplicate keyframes (NDKs) to detect web video topics and Zhang et al. [27] also fuse NDKs with event keywords, but their methods still rely strongly on video tags. Fu et al. [28] proposed a multigraph fusion method that fuses the video shots by a hypergraph for video summarization. Similarly, Chu et al. [29] proposed a multimodal graph fusion method for cross-media topic detection. Multimodal data fused by graphs may overlook semantic differences between modalities and features have no semantic interaction with each other.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe transformer architecture may be an effective tool for fusing multimodal features of video, which has empowered language models to transfer large quantities of data to low-resource domains [30]. And it trains short text data to obtain concise text representation with rich semantics [31]. Rani [32] used the transformer model to fuse heterogeneous data (title, tag and description) for the discovery of web video topics. Zhu [33] adopted a transformer-based encoder-decoder architecture fusing the topical and commonsense features to detect emotion in dialogue text. However, the above studies only fuse textual features, and there are hardly any papers on video topic detection that use transformer-based models to fuse multimodal features.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.2 Topic evolution\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTopic evolution is important for capturing trends in related fields\u003csup\u003e\u0026nbsp;\u003c/sup\u003e[34], and it is often applied to scientific literature or web text. Some studies apply co-word analysis to topic evolution [35-36], which is based on the quantitative calculation of keyword frequency and co-occurrence frequency, and aggregates into topic clusters according to the correlation strength between keyword pairs. It is a common method to investigate topic evolution by measuring the similarity between different topics, and then explore the evolution process between topics [37-38]. In recent years, some scholars believe that topic evolution can be analyzed from multiple dimensions such as structure, intensity and content [39-40]. On this basis, Zhu et al. proposed a \u0026lsquo;word-topic\u0026rsquo; coupling network to analyze the process of topic [41] in order to comprehensively describe the evolution process and state of topics.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eCompared to the field of textual information, there are very few studies on the evolution of video topics. Ryosuke et al. proposed a topic evolution tracking algorithm based on salient keyword matching with consideration of semantic broadness for web videos, and tracked the topic evolution from the hierarchy of video groups [42], which can make users understand the topic evolution over time to find web videos they are interested in, but the calculation cost is high and the amount of data is large. Cao et al. presented an efficient algorithm based on salient trajectory extraction on a topic evolution link graph, which considers tag and visual similarities [43]. There is a lot of work on topic evolution based on text data, and the analytical dimension includes the topic intensity and content, but there is little work on video topic evolution.\u003c/p\u003e\n\u003cp\u003eIn summary, there is limited research on topic analysis for multimedia such as videos. Table 1 summarizes the recent studies on video topic detection and evolution. It can be seen from the table that video features for topic detection can be roughly divided into two categories: single-modality and multimodality of videos. The former just considers the text or keyframes of videos, which means that the obtained topics are not well illustrative. The existing work on video multimodal fusion does not consider the semantic level differences among modalities and there is no semantic interaction between features. In this paper, we convert the different features of videos into high-level textual features with rich semantics, and to realize semantic interactions among multimodalities, the transformer-based model is used to fuse features. The HDBSCAN is applied to cluster videos to obtain topics, and the comparative experiments of different\u0026nbsp;clustering methods, single-modal and multimodal are\u0026nbsp;performed\u0026nbsp;respectively. Furthermore, the evolution of video topics is analyzed from the perspective of intensity and content.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 1.\u0026nbsp;\u003c/strong\u003eComparison of previous studies on the video topic detection and evolution.\u0026nbsp;The letter Y and N represent \u0026lsquo;with and without the referred research point\u0026rsquo; respectively.\u003c/p\u003e\n\u003cdiv\u003e\n \u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"596\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"21.644295302013422%\"\u003e\n \u003cp\u003e\u003cstrong\u003eReferences\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.778523489932887%\"\u003e\n \u003cp\u003e\u003cstrong\u003eFeatures\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"28.187919463087248%\"\u003e\n \u003cp\u003e\u003cstrong\u003eMethods of topic detection\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"17.449664429530202%\"\u003e\n \u003cp\u003e\u003cstrong\u003eContent\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003eevolution\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.939597315436242%\" colspan=\"2\"\u003e\n \u003cp\u003e\u003cstrong\u003eIntensity evolution\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"21.644295302013422%\"\u003e\n \u003cp\u003eLiu et al. [17]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.778523489932887%\"\u003e\n \u003cp\u003eKeywords\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"28.187919463087248%\"\u003e\n \u003cp\u003eBipartite graph model\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"18.120805369127517%\" colspan=\"2\"\u003e\n \u003cp\u003eN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.268456375838927%\"\u003e\n \u003cp\u003eN\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"21.644295302013422%\"\u003e\n \u003cp\u003eLi et al. [23]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.778523489932887%\"\u003e\n \u003cp\u003eKeyframes;\u003c/p\u003e\n \u003cp\u003eTags\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"28.187919463087248%\"\u003e\n \u003cp\u003eSemi-supervised\u003c/p\u003e\n \u003cp\u003emachine learning\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"18.120805369127517%\" colspan=\"2\"\u003e\n \u003cp\u003eN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.268456375838927%\"\u003e\n \u003cp\u003eN\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"21.644295302013422%\"\u003e\n \u003cp\u003eRyosuke et al. [42]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.778523489932887%\"\u003e\n \u003cp\u003eKeywords;\u003c/p\u003e\n \u003cp\u003eKeyframes\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"28.187919463087248%\"\u003e\n \u003cp\u003eKeyword matching\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"18.120805369127517%\" colspan=\"2\"\u003e\n \u003cp\u003eNetwork analysis\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.268456375838927%\"\u003e\n \u003cp\u003eN\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"21.644295302013422%\"\u003e\n \u003cp\u003eCao et al. [43]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.778523489932887%\"\u003e\n \u003cp\u003eTags\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"28.187919463087248%\"\u003e\n \u003cp\u003eEvent Mining;\u003c/p\u003e\n \u003cp\u003eDetection of burst tags\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"18.120805369127517%\" colspan=\"2\"\u003e\n \u003cp\u003eTopic track diagram\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.268456375838927%\"\u003e\n \u003cp\u003eN\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"21.644295302013422%\"\u003e\n \u003cp\u003eRani et al. [32]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.778523489932887%\"\u003e\n \u003cp\u003eText\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"28.187919463087248%\"\u003e\n \u003cp\u003eBERT;\u003c/p\u003e\n \u003cp\u003eHDBSCAN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"18.120805369127517%\" colspan=\"2\"\u003e\n \u003cp\u003eN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.268456375838927%\"\u003e\n \u003cp\u003eN\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"21.644295302013422%\"\u003e\n \u003cp\u003eCurrent study\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.778523489932887%\"\u003e\n \u003cp\u003eText;\u003c/p\u003e\n \u003cp\u003eAudio;\u003c/p\u003e\n \u003cp\u003eCovers\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"28.187919463087248%\"\u003e\n \u003cp\u003eTransformer;\u003c/p\u003e\n \u003cp\u003eHDBSCAN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"18.120805369127517%\" colspan=\"2\"\u003e\n \u003cp\u003eTopic word-based\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.268456375838927%\"\u003e\n \u003cp\u003eTopic river diagram\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e"},{"header":"Methodology","content":"\u003cp\u003eIn this section, we will introduce the detail of the proposed video topic detection method, which is divided into three phases: data acquisition and video representation, topic detection and evolution analysis. Specifically, data acquisition and video representation involve acquiring video metadata, transforming the different modalities into high-level semantic text, and videos are represented by high-dimensional vectors with 1 row and 768 columns like [0.1444 -0.062 \u0026hellip; 0.096 -0.072], which are learned from the fused texts based on the transformer encoder. Topic detection is mainly performed by the HDBSCAN clustering method, and it is compared with hierarchical clustering. We respectively illustrate the evolution of topics in terms of the intensity and content. The intensity is measured by the number of videos and the content evolution is revealed by the change of keywords sorted by TF-IDF. The workflow is shown in Fig. 1 and the proposed method of topic detection is depicted in algorithm 1.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3.1 Preprocessing\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIn order to improve the accuracy and performance of the proposed method, preprocessing is essential for videos\u0026rsquo; titles, tags and description texts, audios and covers. Texts like titles, tags and descriptions are processed by removing invalid symbols, anomalies, null values and then they are combined into one text data. Audios are converted into text by automatic speech recognition (ASR), and the converted audio texts are too long and contain invalid data, so the summary extraction is performed for them. Video covers are applied with optical character recognition (OCR) to detect texts. The process in shown in Fig. 2a.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3.2 Data fusion and embedding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAs shown in Fig. 2a, data fusion aims to integrate multiple modal features of videos to provide a more comprehensive description of videos. Considering the misalignment and semantic level differences among different modalities of videos, the audios and video covers are transformed into the text form with richer semantics. When performing fusion, add separators \u0026lsquo;ESP\u0026rsquo; as a distinction between texts of different modalities, and use \u0026lsquo;CLS\u0026rsquo; as a beginning symbol, i.e., \u0026lsquo;CLS\u0026rsquo; + \u0026lsquo;video title text\u0026rsquo; + \u0026lsquo;ESP\u0026rsquo; + \u0026lsquo;audio text\u0026rsquo; + \u0026lsquo;ESP\u0026rsquo; + \u0026lsquo;video cover text\u0026rsquo; + \u0026lsquo;ESP\u0026rsquo;.\u003c/p\u003e\n\u003cp\u003eVideo embedding is to convert videos into feature vectors that are learned from the multimodalities. We hope that semantic interactions can be generated among the fused features, resulting in semantically richer fused features. The attention mechanism in the transformer model aims to better understand the contextual relationships of words in texts, which means each word is calculated with other words [32], as shown in Fig. 2b. Therefore, the transformer-based model is used for multimodal fusion to represent videos in the same semantic vector space. Then the similarity between videos can be measured by calculating the cosine similarity of the vectors.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3.3 Topic detection\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eConsidering the multimodal features of videos, previous textual topic models cannot be used to detect the video topics, so clustering algorithms are applied for video topic detection in the paper. The clusters are formed by videos with high similarity, and each cluster represents a topic. Since the video vector has high dimensionality, it may not work well in clustering [32]. We firstly reduce the dimensionality and then perform HDBSCAN clustering. The Uniform Manifold Approximation and Projection (UMAP) is employed to achieve it. Compared to some existing dimensionality reduction methods like PCA or t-SNE, UMAP preserves more local and global features of high-dimensional data at a lower projection dimension [44]. And Allaoui et\u0026nbsp;al. [45] demonstrated that the performance of HDBSCAN is improved by reducing high-dimensional embeddings with UMAP, in terms of clustering accuracy and time. Thus, UMAP is used to reduce the dimensionality of embeddings.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eIntroduction of HDBSCAN method.\u003c/strong\u003e HDBSCAN defines a new distance measurement, the mutual reachability distance that can better reflect the density of points and thus it can handle clustering problems with different densities. When performing clustering, HDBSCAN uses a soft-clustering approach allowing noise to be modeled as outliers, which prevents irrelevant data from being assigned to any cluster and is expected to improve the clustering results [46]. Therefore, HDBSCAN is adopted to cluster similar videos to generate topics. It can be roughly divided into these steps: transforming the space, building the minimum spanning tree, building the cluster hierarchy, condensing the cluster tree and extracting the clusters, as shown in Fig. 3.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eIntroduction of HC method.\u0026nbsp;\u003c/strong\u003eHierarchical clustering creates a hierarchical nested clustering tree by calculating the distance between data points of different categories, as shown in Fig. 4a. It does not need to determine the number of clusters in advance, and from the Fig. 4b, it is easy to discover the hierarchical relationship, similar to HDBSCAN. Then the hierarchical clustering method is used for comparison experiments. Using the video similarity as the measure of the distance between videos, the initial video distance matrix is constructed. The distance matrix is updated after each iteration until no new cluster appears, where the distance between different clusters is calculated by average distance.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3.4 Topic evolution\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe state of topic evolution can be represented by topic intensity and content. Topic intensity can be regarded as the degree of attention and discussion in a period of time [39]. The more the number of blogs to discuss a topic, the higher the hotness of the topic is [38]. For the video topics represented by clusters, the most intuitive way of measuring the topic intensity is the cluster size. When analyzing the evolution of topic content, we can analyze the change of topic words in different time slices. The fused texts of videos in clusters are segmented, and the words are sorted by TF-IDF to get the top ranked keywords. Then the content evolution can be explained based on the change of keywords.\u003c/p\u003e"},{"header":"Experiments and results","content":"\u003cp\u003e\u003cstrong\u003e4.1 Experimental dataset and preprocessing\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe crawl 2067 videos from the issue 147 to issue 192 of the \u0026ldquo;Weekly Must-see\u0026rdquo; series on the Bilibili platform, spanning from Jan. 7th, 2022 to Nov. 24th, 2022. In this case, one issue of videos refers to the videos published in one week, so we take two issues (half a month) as a time unit. Each video\u0026rsquo;s titles, tags and descriptions, cover and audio are crawled and saved.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eBefore topic detection, it is necessary to preprocess the input data. We remove the invalid symbols and null values from the video textual data and combine the processed text. The audios are converted to text by\u0026nbsp;automatic speech recognition\u0026nbsp;and some inaccurate converted texts are corrected manually. The texts in the covers are also recognized by\u0026nbsp;optical character recognition. Then we fuse the data in different modalities and input them into the transformer-based model to get video vectors.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e4.2 Performance evaluation\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIn this paper, video topics are detected by clustering methods, which is an unsupervised process. In order to better evaluate and compare the experimental results, Silhouette coefficient (as shown in Equation (1), (2)) is adopted as the experimental evaluation metric, which is a way to evaluate how good the clustering effect is. The larger the Silhouette coefficient, the greater the distance between the cluster structure where the sample point is located and its nearest cluster structure, thus indicating a better clustering result at this point.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"https://myfiles.space/user_files/122228_c8a1650c59388082/122228_custom_files/img1712174627.png\"\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cp\u003eIn order to testify the proposed method, two main comparative experiments are conducted. One is the comparison of clustering methods, and the other is the comparison of different numbers of modalities. The silhouette coefficient of various approaches has been presented in Fig. 5a and Fig. 5b.\u003c/p\u003e\n\u003col class=\"decimal_type\"\u003e\n \u003cli\u003eFrom the Fig. 5a, it can be concluded that the HDBSCAN is better than the method of HC when the number of clusters is greater than four. And the difference is getting bigger as the number of clusters increases.\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eFurthermore, Fig. 5b shows that the proposed transformer-based multimodal fusion method outperforms single-modal\u0026nbsp;when detecting video topics by HDBSCAN.\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eAccording to the above comparison results, we can see that HDBSCAN can be used to mine video topics and performs well in the case of multimodal fusion, so it is necessary to make full use of the multimodal features.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e4.3 Analysis of topic evolution\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe state evolution of video topics is analyzed from the perspective of intensity and content respectively. The topics that appear in every time unit (half a month) are selected as the hot topic, and part of hot topics are shown in the Table 2. The topic evolution with a period of three and a half months is analyzed, where the intensity of topics is measured by the number of videos in each cluster and the video content is examined through the keywords sorted by TF-IDF.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 2.\u0026nbsp;\u003c/strong\u003eThe top 10 keywords of each hot topic in issue 179 to issue 180.\u0026nbsp;The \u0026lsquo;Topic\u0026rsquo; column is the id of hot topics. The \u0026lsquo;Video numbers\u0026rsquo; column is\u0026nbsp;the number of videos in each cluster.\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"548\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"8.394160583941606%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003eTopic\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"59.12408759124087%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003eTop 10 keywords\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.970802919708028%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003eVideo numbers\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.510948905109489%\" valign=\"top\"\u003e\n \u003cp\u003e\u003cstrong\u003eTopic name\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"8.394160583941606%\" valign=\"top\"\u003e\n \u003cp\u003eT\u003csub\u003e0\u003c/sub\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"59.12408759124087%\" valign=\"top\"\u003e\n \u003cp\u003eThe games; The Black Myth; Game videos; Single-player games; LOL; Players; Sandbox games; Wukong; Live demo; Minecraft,\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.970802919708028%\" valign=\"top\"\u003e\n \u003cp\u003e10\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.510948905109489%\" valign=\"top\"\u003e\n \u003cp\u003eGame\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"8.394160583941606%\" valign=\"top\"\u003e\n \u003cp\u003eT\u003csub\u003e3\u003c/sub\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"59.12408759124087%\" valign=\"top\"\u003e\n \u003cp\u003eChinese music;\u0026nbsp;Songs;\u0026nbsp;Originals; MV; Demo; High pitch;\u0026nbsp;Music creation; Music;\u0026nbsp;Classic Songs; Languages\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.970802919708028%\" valign=\"top\"\u003e\n \u003cp\u003e3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.510948905109489%\" valign=\"top\"\u003e\n \u003cp\u003eMusic\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"8.394160583941606%\" valign=\"top\"\u003e\n \u003cp\u003eT\u003csub\u003e4\u003c/sub\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"59.12408759124087%\" valign=\"top\"\u003e\n \u003cp\u003eXu Mi; Genshin Impact; Version; Music; Models; Mobile Games;\u0026nbsp;Dubbing;\u0026nbsp;God\u0026rsquo;s pupil;\u0026nbsp;Scenes; Zhong Li\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.970802919708028%\" valign=\"top\"\u003e\n \u003cp\u003e8\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.510948905109489%\" valign=\"top\"\u003e\n \u003cp\u003eGenshin Impact\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"8.394160583941606%\" valign=\"top\"\u003e\n \u003cp\u003eT\u003csub\u003e5\u003c/sub\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"59.12408759124087%\" valign=\"top\"\u003e\n \u003cp\u003eMake-up;\u0026nbsp;Ancient style;\u0026nbsp;Red lips;\u0026nbsp;Man in the painting;\u0026nbsp;Pollen;\u0026nbsp;Dream of the red Mansions; Cos;\u0026nbsp;Sexy girl;\u0026nbsp;Immortal Sword;\u0026nbsp;Outfit matching\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.970802919708028%\" valign=\"top\"\u003e\n \u003cp\u003e3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.510948905109489%\" valign=\"top\"\u003e\n \u003cp\u003eMake-up and clothing\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"8.394160583941606%\" valign=\"top\"\u003e\n \u003cp\u003eT\u003csub\u003e6\u003c/sub\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"59.12408759124087%\" valign=\"top\"\u003e\n \u003cp\u003eAuto-tune remix-themed content; Cai Xukun; Chicken; Ikun;\u0026nbsp;Planning;\u0026nbsp;Section 3;\u0026nbsp;Fierce man;\u0026nbsp;Cars;\u0026nbsp;Learning to drive;\u0026nbsp;Liu Huaqiang\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.970802919708028%\" valign=\"top\"\u003e\n \u003cp\u003e4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.510948905109489%\" valign=\"top\"\u003e\n \u003cp\u003eAuto-tune remix-themed content\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"8.394160583941606%\" valign=\"top\"\u003e\n \u003cp\u003eT\u003csub\u003e7\u003c/sub\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"59.12408759124087%\" valign=\"top\"\u003e\n \u003cp\u003eFood;\u0026nbsp;Midsummer;\u0026nbsp;Tutorial;\u0026nbsp;Cookery;\u0026nbsp;Seafood;\u0026nbsp;Chinese style;\u0026nbsp;Lobsters;\u0026nbsp;Fresh milk;\u0026nbsp;Food ingredients;\u0026nbsp;Cartilaginous fish\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.970802919708028%\" valign=\"top\"\u003e\n \u003cp\u003e10\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.510948905109489%\" valign=\"top\"\u003e\n \u003cp\u003eFood\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"8.394160583941606%\" valign=\"top\"\u003e\n \u003cp\u003eT\u003csub\u003e11\u003c/sub\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"59.12408759124087%\" valign=\"top\"\u003e\n \u003cp\u003eAnimation;\u0026nbsp;Comics; Tom \u0026amp; Jerry; Seminar;\u0026nbsp;Carnival;\u0026nbsp;Styles;\u0026nbsp;Dubbing;\u0026nbsp;Variety comics;\u0026nbsp;Animation News;\u0026nbsp;Painter;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.970802919708028%\" valign=\"top\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.510948905109489%\" valign=\"top\"\u003e\n \u003cp\u003eComics and animation\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"8.394160583941606%\" valign=\"top\"\u003e\n \u003cp\u003eT\u003csub\u003e12\u003c/sub\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"59.12408759124087%\" valign=\"top\"\u003e\n \u003cp\u003eDancing; Dance moves; Choreography; Divine comedy;\u0026nbsp;Original song;\u0026nbsp;Double dance;\u0026nbsp;Vocaloid; Cosplay; Manpower;\u0026nbsp;Roles\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.970802919708028%\" valign=\"top\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.510948905109489%\" valign=\"top\"\u003e\n \u003cp\u003eDancing\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"8.394160583941606%\" valign=\"top\"\u003e\n \u003cp\u003eT\u003csub\u003e13\u003c/sub\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"59.12408759124087%\" valign=\"top\"\u003e\n \u003cp\u003eHigh energy;\u0026nbsp;Little theatre;\u0026nbsp;Magic;\u0026nbsp;Dark Humor;\u0026nbsp;variety show;\u0026nbsp;Talent show;\u0026nbsp;Brainwashing;\u0026nbsp;Brainstorming;\u0026nbsp;Parody show; Funny\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.970802919708028%\" valign=\"top\"\u003e\n \u003cp\u003e7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.510948905109489%\" valign=\"top\"\u003e\n \u003cp\u003eFunny videos\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"8.394160583941606%\" valign=\"top\"\u003e\n \u003cp\u003eT\u003csub\u003e15\u003c/sub\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"59.12408759124087%\" valign=\"top\"\u003e\n \u003cp\u003eMovies;\u0026nbsp;High scores;\u0026nbsp;Emmy Award;\u0026nbsp;American drama;\u0026nbsp;Movie box office; television play;\u0026nbsp;flow rate;\u0026nbsp;Game of Thrones;\u0026nbsp;Film and television;\u0026nbsp;famous scene\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.970802919708028%\" valign=\"top\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.510948905109489%\" valign=\"top\"\u003e\n \u003cp\u003eFilm and television\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"8.394160583941606%\" valign=\"top\"\u003e\n \u003cp\u003eT\u003csub\u003e16\u003c/sub\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"59.12408759124087%\" valign=\"top\"\u003e\n \u003cp\u003eTechnology products; Digital;\u0026nbsp;Cell phone;\u0026nbsp;Software;\u0026nbsp;Cooling;\u0026nbsp;Overseas;\u0026nbsp;Domestic mobile phones; App; Records\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"16.970802919708028%\" valign=\"top\"\u003e\n \u003cp\u003e4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"15.510948905109489%\" valign=\"top\"\u003e\n \u003cp\u003eTechnology product reviews\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eA topic river diagram is used to visualize the evolution of video topic intensity, as shown in Fig. 6. There are differences in intensity among hot topics in each time unit, like the hot topic of \u0026lsquo;game\u0026rsquo; and \u0026lsquo;make-up and clothing\u0026rsquo;. The intensity of each video topic is not necessarily constant, and it has fluctuations over time. For instance, the topic of \u0026lsquo;Genshin Impact\u0026rsquo; has some hotness on September 22, then increases significantly on October 8, but decreases again after November 3. Through the topic river diagram, it is possible to clearly understand the intensity difference among hot topics and examine the intensity change of hot video topics over time.\u003c/p\u003e\n\u003cp\u003eThe fused multimodal features are finally presented in the form of text with richer semantics, so in order to reflect the content evolution of the video topics, the text in the clusters can be divided into words, and the words under each topic are sorted by TF-IDF. The evolution of \u0026lsquo;game\u0026rsquo; topic from issue 179 to issue192 (three and a half months) is reflected by keywords in the Fig. 7, which shows the top ten keywords ranked by TF-IDF. It can be seen that there exist changes in the content of topic \u0026lsquo;game\u0026rsquo; over different periods. Its keywords are not exactly the same in the whole period, which is related to some online hot events. For example, the keywords of \u0026lsquo;LPL\u0026rsquo;, \u0026lsquo;LOL\u0026rsquo; and \u0026lsquo;S12\u0026rsquo; are contained in issue 181 to issue 184, because the LOL\u0026rsquo;s 12th global tournament was held in that period. By examining the changes in keywords, the evolution of video topics can be analyzed.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eTopic detection and tracking in video domain is very interesting. In this paper we propose to use the transformer-based model to fuse multimodal features for detecting video topics. The method fully takes into account the semantic level differences among multimodalities, and enables semantic interaction between features to obtain fused multimodal features with richer semantics. Additionally, the intensity and content evolution of video topics are analyzed based on the river diagram and topic keywords respectively.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eExperimental results show that the HDBSCAN method is better than the hierarchical clustering in video topic detection. The HDBSCAN just need to set a few parameters while the hierarchical clustering requires setting division thresholds which is susceptible to subjective factors. For video topic detection, the method of fusing multimodal features is better than that of single modality, so it is necessary to make full use of videos\u0026rsquo; multimodal features. Analyzing the evolution of video topics in terms of intensity and content allows us to understand the state change of topics on a macro and micro level.\u003c/p\u003e\n\u003cp\u003eThe detection and evolution analysis of video topics are conducted through modeling and experiments, and it can keep users informed of online video topics and aid the platform to manage videos with related topics. Furthermore, it is also meaningful for video recommendation, and even for predicting hot topics of videos.\u003c/p\u003e\n\u003cp\u003eAlthough the current study provides a discussion on hot topic detection and evolution analysis of web videos, it inevitably has its own limitations. The topic detection is based on unsupervised clustering methods, which depends on the setting of parameters, and in the future, self-supervised learning methods that learn the video information from labelled data, can be applied for video topic detection to improve the robustness of the model. In addition, we have fused modalities such as the title, audio and cover of videos, and the video content is also the part of video topics, which may be considered in future research.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe are very thankful to the editor and people whose valuable suggestions and comments helped us to improve the article.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDeclaration of conflicting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis article is supported by the National Natural Science Foundation of China under Grant 72374111, 71874088, and the Postgraduate Research and Practice Innovation Project in Jiangsu Province under the grant KYCX22_0876.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003cstrong\u003eData Availability Statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe data that support the findings of this study are available from the corresponding author, upon reasonable request.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eXie L, Natsev A, Kender J R, et al. Visual memes in social media: tracking real-world news in youtube videos. Proceedings of the 19th ACM international conference on Multimedia. 2011: 53-62. https://doi.org/10.1145/2072298.2072307.\u003c/li\u003e\n\u003cli\u003eRan L, Suzhi X, Yuanyuan R, et al. A modified approach of hot topics found on micro-blog. Frontier and Future Development of Information Technology in Medicine and Education. Springer, Dordrecht, 2014: 603-614. https://doi.org/10.1007/978-94-007-7618-0_58.\u003c/li\u003e\n\u003cli\u003eChen K., Luesukprasert L., Chou, S. Hot topic extraction based on timeline analysis and multidimensional sentence modeling, IEEE transactions on knowledge and data engineering. 19 (8) (2007) 1016-1025. 10.3969/j.issn.1005-8095.2020.11.019.\u003c/li\u003e\n\u003cli\u003eDaouadi K., Reba\u0026iuml; R., Amous I. Optimizing semantic deep forest for tweet topic classification, Information Systems. 101 (2001) 101801. https://doi.org/10.1016/j.is.2021.101801.\u003c/li\u003e\n\u003cli\u003eBlei D M, Ng A Y, Jordan M I. Latent dirichlet allocation. Journal of Machine Learning Research, 2003, 3(Jan):993-1022.\u003c/li\u003e\n\u003cli\u003eK.H. Lim, A. Datta. Following the follower: Detecting communities with common interests on Twitter. In: Proceedings of the 23rd ACM conference on Hypertext and social media, Milwaukee, Wisconsin, USA, 25-28 June 2012, pp. 317-318. https://doi.org/10.1145/2309996.2310052.\u003c/li\u003e\n\u003cli\u003eXie W, Zhu F, Jiang J et al. Topic sketch: Real-time bursty topic detection from twitter, IEEE Transactions on Knowledge and Data Engineering. 28 (8) (2016) 2216-2229. 10.1109/TKDE.2016.2556661.\u003c/li\u003e\n\u003cli\u003eLu Y, Zhang P, Liu J et al. Health-related hot topic detection in online communities using text clustering, Plos one. 8 (2) (2013) e56221. https://doi.org/10.1371/journal.pone.0056221.\u003c/li\u003e\n\u003cli\u003ePons A, Berlanga R, Ruiz J. Topic discovery based on text mining techniques, Information processing \u0026amp; management. 43 (3) (2007) 752-768. https://doi.org/10.1016/j.ipm.2006.06.001.\u003c/li\u003e\n\u003cli\u003eZhao F, Zhu Y, Jin effeeffectH et al. A personalized hashtag recommendation approach using LDA-based topic model in microblog environment, Future Generation Computer Systems. 65 (2016) 196-206. https://doi.org/10.1016/j.future.2015.10.012.\u003c/li\u003e\n\u003cli\u003eGuo X, Xiang Y, Chen Q et al. LDA-based online topic detection using tensor factorization, Journal of information science. 39 (4) (2013) 459-469. https://doi.org/10.1177/0165551512473066.\u003c/li\u003e\n\u003cli\u003eAlSumait L, Barbar\u0026aacute; D, Domeniconi C. On-line lda: Adaptive topic models for mining text streams with applications to topic detection and tracking. In: 2008 eighth IEEE international conference on data mining, Pisa, Italy,15-19 December 2008, pp. 3-12. https://doi.org/10.1109/ICDM.2008.140.\u003c/li\u003e\n\u003cli\u003eChen L C. An effective LDA-based time topic model to improve blog search performance. Information Processing \u0026amp; Management, 2017, 53(6): 1299-1319. https://doi.org/10.1016/j.ipm.2017.08.001.\u003c/li\u003e\n\u003cli\u003eLi W, Feng Y, Li D, et al. Micro-blog topic detection method based on BTM topic model and K-means clustering algorithm. Automatic Control and Computer Sciences, 2016, 50(4): 271-277. https://doi.org/10.3103/S0146411616040040.\u003c/li\u003e\n\u003cli\u003eGargi U, Lu W, Mirrokni V et al. Large-scale community detection on youtube for topic discovery and exploration. In: Fifth International AAAI Conference on Weblogs and Social Media, Barcelona, Catalonia, Spain, 17-21 July 2011, pp. 486-489. http://www.aaai.org/ocs/index.php/ICWSM/ICWSM11/paper/view/2864.\u003c/li\u003e\n\u003cli\u003eCao J, Zhang Y, Ji R et al. Web video topics discovery and structuralization with social network, Neurocomputing. 172 (2016) 53-63. https://doi.org/10.1016/j.neucom.2014.10.103.\u003c/li\u003e\n\u003cli\u003eLiu L, Sun L, Rui Y et al. Web video topic discovery and tracking via bipartite graph reinforcement model. In: Proceedings of the 17th international conference on World Wide Web, Beijing, China, April 2008, pp. 1009-1018. https://doi.org/10.1145/1367497.1367633.\u003c/li\u003e\n\u003cli\u003eWang Y, Wu T, Li G et al. Video topic detection on Micro-Blog using Relational Topic Model, Academic Journal of Computing \u0026amp; Information Science. 4 (2) (2021). https://doi.org/10.25236/AJCIS.2021.040214.\u003c/li\u003e\n\u003cli\u003eLiu Y, Yu N. Dual linkage refinement for YouTube video topic discovery. In: 2010 IEEE International Conference on Multimedia and Expo, Singapore, 19-23 July 2010, pp. 1576-1581. 10.1109/ICME.2010.5582943.\u003c/li\u003e\n\u003cli\u003eLiu K, Xu J, Zhang L et al. Discovering hot topics from geo-tagged video, Neurocomputing. 105(2013) 90-99. https://doi.org/10.1016/j.neucom.2012.05.035.\u003c/li\u003e\n\u003cli\u003eBian J, Huang M L. Semantic topic discovery for lecture video. In: Proceedings of SAI Intelligent Systems Conference, London, UK, September 2019, pp.457-466. https://doi.org/10.1007/978-3-030-29516-5_36.\u003c/li\u003e\n\u003cli\u003eZuo L. Research on the automatic generation scheme of interview short video titles, Computer Products and Circulation. 11(2017) 158. CNKI:SUN:WXXJ.0.2017-11-144.\u003c/li\u003e\n\u003cli\u003eLi G, Jiang S, Zhang W et al. Online web video topic detection and tracking with semi-supervised learning, Multimedia Systems. 22 (1) (2016) 115-125. https://doi.org/10.1007/s00530-014-0402-0.\u003c/li\u003e\n\u003cli\u003eShao J, Ma S, Lu W et al. A unified framework for web video topic discovery and visualization, Pattern Recognition Letters. 33 (4) (2012) 410-419. https://doi.org/10.1016/j.patrec.2011.07.026.\u003c/li\u003e\n\u003cli\u003eShao J, Yin W, Ma S et al. Topic discovery of web video using star-structured k-partite graph. In: Proceedings of the 18th ACM international conference on Multimedia, Firenze, Italy, October 2010, pp. 915-918. https://doi.org/10.1145/1873951.1874112.\u003c/li\u003e\n\u003cli\u003eChen T, Liu C, Huang Q. An effective multi-clue fusion approach for web video topic detection. Proceedings of the 20th ACM international conference on Multimedia. 2012: 781-784. https://doi.org/10.1145/2393347.2396311.\u003c/li\u003e\n\u003cli\u003eZhang C, Wu X, Shyu M L, et al. Adaptive association rule mining for web video event classification. 2013 IEEE 14th International Conference on Information Reuse \u0026amp; Integration (IRI). IEEE, 2013: 618-625. https://doi.org/10.1109/IRI.2013.6642526.\u003c/li\u003e\n\u003cli\u003eFu Y, Guo Y, Zhu Y, et al. Multi-view video summarization. IEEE Transactions on Multimedia, 2010, 12(7): 717-729. https://doi.org/10.1109/TMM.2010.2052025.\u003c/li\u003e\n\u003cli\u003eChu L, Zhang Y, Li G, et al. Effective multimodality fusion framework for cross-media topic detection. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 26(3): 556-569. https://doi.org/10.1109/TCSVT.2014.2347551.\u003c/li\u003e\n\u003cli\u003eAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017(30), pages 5998\u0026ndash;6008.\u003c/li\u003e\n\u003cli\u003eCheng P, Du J, Kou F, et al. Topic Detection Based on Semantics, Time and Social Relationship. Proceedings of 2019 Chinese Intelligent Automation Conference. Springer Singapore, 2020: 691-698. https://doi.org/10.1007/978-981-32-9050-1_78.\u003c/li\u003e\n\u003cli\u003eRani S, Kumar M. Heterogeneous Information Fusion based Topic Detection from Social Media Data. Information Systems Frontiers, 2022: 1-16. https://doi.org/10.1007/s10796-022-10334-w.\u003c/li\u003e\n\u003cli\u003eZhu L, Pergola G, Gui L, et al. Topic-driven and knowledge-aware transformer for dialogue emotion detection. arXiv preprint arXiv:2106.01071, 2021. https://doi.org/10.48550/arXiv.2106.01071.\u003c/li\u003e\n\u003cli\u003eChen B, Tsutsui S, Ding Y et al. Understanding the topic evolution in a scientific domain: An exploratory study for the field of information retrieval, Journal of Informetrics. 11 (4) (2017) 1175-1189. https://doi.org/10.1016/j.joi.2017.10.003.\u003c/li\u003e\n\u003cli\u003eTang G, Zhang W. Research progress and analysis of subject evolution based on co-word analysis method, Library and Information Service. 59 (5) (2015) 128-136. 10.13266/j.issn.0252-3116.2015.05.020.\u003c/li\u003e\n\u003cli\u003eWei L, Jiamin W, Jiming H. Analyzing the topic distribution and evolution of foreign relations from parliamentary debates: A framework and case study, Information Processing \u0026amp; Management. 57 (3) (2020) 102191. https://doi.org/10.1016/j.ipm.2019.102191.\u003c/li\u003e\n\u003cli\u003eLv N, Luo J, Liu Y et al. Analysis of topic evolution based on subtopic similarity. In: 2009 International Conference on Computational Intelligence and Natural Computing, Wuhan, China, 6-7 June 2009, pp. 506-509. 10.1109/CINC.2009.23.\u003c/li\u003e\n\u003cli\u003eJian F, Ya W, Yuan D. Microblog topic evolution computing based on LDA algorithm, Open Physics. 16 (1) (2018) 509-516. https://doi.org/10.1515/phys-2018-0067.\u003c/li\u003e\n\u003cli\u003eLiu Z, Wang X, Bai R. Construction and Empirical Research on Multi-dimensional Topic Evolution Analysis Model, Information Theory and Practice. 40 (3) (2017) 92-98. 10.16353/j.cnki.1000-7490.2017.03.018.\u003c/li\u003e\n\u003cli\u003eWang X, Cheng Q, Lu W. Analyzing evolution of research topics with NEViewer: a new method based on dynamic co-word networks, Scientometrics. 101 (2) (2014) 1253-1271. https://doi.org/10.1007/s11192-014-1347-y.\u003c/li\u003e\n\u003cli\u003eZhu, H., Qian, L., Qin, W. et al. Evolution analysis of online topics based on \u0026lsquo;word-topic\u0026rsquo; coupling network. Scientometrics 127, 3767\u0026ndash;3792 (2022). https://doi.org/10.1007/s11192-022-04439-x\u003c/li\u003e\n\u003cli\u003eHarakawa R, Ogawa T, Haseyama M. Tracking topic evolution via salient keyword matching with consideration of semantic broadness for Web video discovery, Multimedia Tools and Applications. 77 (16) (2018) 20297-20324. https://doi.org/10.1007/s11042-017-5404-4.\u003c/li\u003e\n\u003cli\u003eCao J, Ngo C, Zhang Y et al. Tracking web video topics: Discovery, visualization, and monitoring, IEEE Transactions on Circuits and Systems for Video. Technology. 21 (12) (2011) 1835-1846. 10.1109/TCSVT.2011.2148470.\u003c/li\u003e\n\u003cli\u003eMcInnes, L., Healy, J., \u0026amp; Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv:1802.03426, 2018. https://doi.org/10.48550/arXiv.1802.03426.\u003c/li\u003e\n\u003cli\u003eAllaoui M, Kherfi M L, Cheriet A. Considerably improving clustering algorithms using UMAP dimensionality reduction technique: a comparative study. Image and Signal Processing: 9th International Conference, ICISP 2020, Marrakesh, Morocco, June 4\u0026ndash;6, 2020, 2020: 317-325. https://doi.org/10.1007/978-3-030-51935-3_34.\u003c/li\u003e\n\u003cli\u003eGrootendorst M. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794, 2022. https://doi.org/10.48550/arXiv.2203.05794.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Multimodality fusion, Topic detection, Topic evolution, Transformer, Web videos, Bilibili platform","lastPublishedDoi":"10.21203/rs.3.rs-4190241/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4190241/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eDue to the prevalence of video social media and the increase of user generated content, the Internet is filled with a large amount of unstructured data. Videos often contain multimodal data such as title, tags, images and audios. Therefore, fusion of multimodal features is a valid way for video topic detection. The titles and tags of videos are short and sparse, and they are high level semantics, whereas the audio and images of videos are low level semantics. It is not suitable to represent a video by directly fusing these features. To address the issue, an effective multimodal fusion method based on the transformer model is proposed for detecting video topics. First, video data is crawled from Bilibili platform, and the titles, tags and descriptions of videos are processed by deleting invalid symbols and null values. The audios are converted to text and texts are recognized from video covers. Second, the transformer-based model is applied to fuse the three forms of text from different modalities to represent videos with multi-dimensional vectors. Then the HDBSCAN and hierarchical clustering (HC) are compared by Silhouette coefficient when clustering videos for topic detection. In addition, we compare video topic clustering under multimodal and single-modal. Finally, the intensity and content evolution of video topics over time are analyzed in the paper. Experimental results with the real data collected from Bilibili verify the effectiveness of the proposed method for video topic detection and evolution.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e","manuscriptTitle":"Multimodality Fusion based Topic Detection and Evolution Analysis of Web Videos","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-04-04 19:31:44","doi":"10.21203/rs.3.rs-4190241/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d15c99f8-716f-4a7b-93e9-abe69cd3702e","owner":[],"postedDate":"April 4th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-04-16T18:58:44+00:00","versionOfRecord":[],"versionCreatedAt":"2024-04-04 19:31:44","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4190241","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4190241","identity":"rs-4190241","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}