Joint Enhancement and Classification Constraints for Noisy Speech Emotion Recognition

Abstract In the natural environment, the received speech signal is often interfered by noise, which reduces the performance of speech emotion recognition (SER) system. To this end, a noisy SER method based on joint constraints, including enhancement constraint and arousal-valence classification constraint (EC-AVCC), is proposed. This method extracts multi-domain statistical feature (MDSF) to input the SER model based on joint EC-AVCC using convolution neural network and long short-term memory-attention (CNN-ALSTM). The model is jointly constrained by speech enhancement (SE) and arousal-valence classification (AVC) to get robust features suitable for SER in noisy environment. Besides, in the auxiliary SE task, a joint loss function simultaneously constrains the error of ideal ratio mask and the error of the corresponding MDSF to obtain more robust features. The proposed method does not need to carry out noise reduction preprocessing. Under the joint constraints, it can obtain robust and discriminative deep emotion features, which can improve the emotion recognition performance in noisy environment. The experimental results on the CASIA and EMO-DB datasets show that compared with the baseline, the proposed method improves the accuracy of SER in white noise and babble noise by 4.7%-9.9%.