SDAM: Semantic Annotation Model for Multi-modal Short Videos Based on Deep Neural Network

Video content analysis has a variety of application scenarios, including automatic content-based indexing and retrieval, video surveillance content description, automatic addition of dialogue subtitles for video, etc., which are urgently needed by enterprises and governments. These application scenarios currently require a large amount of manpower to label the specific content of the video, training the relevant personnel will cost a lot of resources, and the efficiency of manual labeling is low, if we can use computers to automatically analyze the video content will save a lot of costs for the relevant industries and bring revolutionary development. Traditional video analysis methods mainly rely on manually constructed features for matching to achieve the coarse classification of video, often dividing the video into a few categories or a dozen categories simply, which cannot be in line with the era of big data. Therefore, how to use the existing data resources and technical means to complete the content analysis of videos gradually becomes the focus of research. For the characteristics of large data volume and real-time of short videos, fast extraction of key frames containing the main information is the direction to focus on. In this paper, we will consider the characteristics of short video multi-modality and fully exploit the information of each modality of short video. We focus on the semantic annotation methods for short videos. To address the problem that traditional video analysis methods cannot obtain high-value semantic information within the video, this paper first obtains key frames from the video that represent the main information of the video by key frame extraction technique. Based on migration learning, we use pre-trained models to improve the target detection and image description algorithms, extract the high-value regions in key frames, and perform semantic description. The research can better address the needs of short video content analysis and provide data basis for subsequent deep application scenarios.