Granularity Based Inter and Intra-Modal Fusion Network for Sarcasm Detection

Abstract Multi-modal sarcasm detection is a task that involves detecting and identifyingsarcasm using multiple modalities of information. The key aspect of this task lies in how to model intra and inter-modality incongruity. Existing multi-modal sarcasm detection methods often focus on incongruity between modalities while overlooking the potential of fully exploring the semantic information within each modality. We tackle the problem in this paper by designing the Granularity Based Inter and Intra-Modal Fusion Network (GIIFN). Our approach combines tradi-tional image algorithms with deep learning models to extract comprehensive andrich semantic information from images. By incorporating a pre-trained languagemodel, we leverage knowledge from large-scale textual data to enhance the anal-ysis of images. Furthermore, our feature interaction model can effectively fuse and interact features at different granularities, capturing fine details and contex-tual information in the images. Through extensive experimental validation, we demonstrate the outstanding performance of our approach in multimodal satiredetection tasks, surpassing existing methods and outperforming state-of-the-art results.