Multi Attention Based Approach for DeepFake Face and Expression Swap Detection and Localization

Abstract Digital media forensics focuses heavily on the detection of altered photos and videos. The likelihood of an image being altered is often determined by a detection approach using binary classi cation. The key problem is identifying the parts that have been altered (i.e., performing segmentation). We propose an attention-based approach to improve and process feature maps for the classi cation and localization tasks. The learned attention maps show the areas that contain useful information and give a visual representation of the manipulated regions, which improves the binary classi cation (real face vs. fake face). The information received from one manipulation localization is shared with the classi cation network, thereby enhancing the performance of both tasks. A localized map highlighting the DeepFake manipulation type is generated by the network's encoder and attention-based decoder. Instead of using encoded spatial data for classi cation, we used features from the rst layer of the decoder, which includes attention-based localized features associated with each DeepFake manipulation. These features are then coupled with frequency domain features to generate a discriminative representation for identifying DeepFakes. Through comprehensive experiments on face-re-enactment and face- swapping datasets, we show that our method is better than plain vanilla binary classi ers., as well as its capacity to manage the mismatch condition for unseen manipulation and attain state-of-the-art performance. We demonstrate that using multiple attentional blocks improves both face forgery detection and manipulated region localization.