Global-Local combined features to detect pain intensity from facial expression images with attention mechanism

Abstract Background: The estimation of pain intensity is critical for medical diagnosis and treatment of patients. However, the current clinical pain assessment methods mainly rely on patients’ self-report and doctors’ assessment, and the assessment results obtained by these two methods are relatively subjective. Recently, with the development of the image monitoring technology and artificial intelligence, automatic pain assessment based on facial expression and behavioral analysis shows potential values in clinical applications. We proposed a convolutional neural network with global and local attention mechanism (GLA-CNN) framework for the effective detection of pain intensity at four level thresholds using facial expression images. Methods: We proposed a convolutional neural network with global and local attention mechanism (GLA-CNN) framework for the effective detection of pain intensity at four level thresholds using facial expression images. The GLA-CNN includes two modules, namely global attention network (GANet) and local attention network (LANet), and the attention mechanism is applied to GANet and LANet respectively. The LANet is responsible for extracting representative local patch features of the face. The GANet extracts pain features from the whole face to compensate for the ignored correlative features between patches in LANet. In the end, the global correlative features and local subtle features are fused for final estimation of pain intensity. Our data were derived from a publicly available UNBC-McMaster dataset. Results: The objective evaluation of pain can be realized effectively by extracting the facial expression features of patients. Our proposed GLA-CNN model achieved 56.45% accuracy in four-level pain assessment, significantly higher than the chance level (25%). Conclusion: Compared with other models, the proposed GLA-CNN model has higher accuracy. This model could assist clinicians to evaluate patients' pain status and give them analgesic programs in time.