Combining episodic and semantic memory: a brain-inspired approach for video captioning

Abstract Traditional video captioning algorithms mainly rely on detecting objects in the video, realizing the simulation of episodic memory. However, these methods neglect the discussion of the implicit relation between objects, lacking the simulation of semantic memory. In this paper, we discuss the mechanisms of episodic and semantic memory and their relations in the human brain, and a semantic memory branch is added. By using a semantic graph linking to an external knowledge base, the interaction mechanism between episodic and semantic memory is modeled to enhance and inhibit the relations between the semantics of the learned video. We evaluate the effectiveness of our method on public benchmarks (Microsoft Research Video Description Corpus (MSVD) and Microsoft Research Video to Text (MSRVTT)). Extensive experiments demonstrate that our model outperforms general state-of-the-art methods and effectively improves the effect of content understanding in videos.