AU-guided Feature Aggregation for Micro-Expression Recognition

Abstract Micro-expressions (MEs) are spontaneous and transient facial movements that reflect real internal emotions and they have been widely applied in various fields. Recent deep learning-based methods have been rapidly developing in the area of micro-expression recognition (MER), but it is typical to focus on one-sided nature of MEs, covering only representational features or low-ranking Action Unit (AU) features. The subtle changes in MEs characterize its feature representation to be weak and inconspicuous, which makes it tough to analyze MEs only from single piece or a small amount of information to achieve a considerable recognition effect. In addition, the lower-order information can only distinguish MEs from a single low-dimensional perspective and neglects the potential of corresponding MEs and AU combinations to each other. To address these issues, we first explore that the higher-order relations of different AU combinations have correspondence with MEs through statistical analysis. Afterwards, based on this attribute, we propose an end-to-end multi-stream model that integrates global feature learning and local muscle movement representation guided by AU semantic information. The comparative experiments were carried out on benchmark datasets, with better performance than the state-of-art methods. Also, the ablation experiments demonstrate the necessity of our model to introduce the information of AU and its relationship to MER.