Learning Subtle Variances on Temporal Sequence with Inception Attention for Skeleton-based Action Recognition

Abstract Graph convolutional network (GCN) has been employed in the task of skeleton-based action recognition with excellent performance. Studies applying GCN on the spatial-temporal graph paid more attention to feature transfer in the sparse spatial space and the capture of the long-term dependence of each spatial node in the time dimension. However, the sequential treatment of spatial and temporal information is inadequate to capture the intimate spatial-temporal dependencies. In addition, the high-resolution temporal information in the spatial-temporal graph has not been explored adequately. For the reasons, we propose to learn subtle variances on temporal sequence based on GCN with Inception Attention for skeleton-based action recognition. To learn the subtle variances of different actions, a self-supervised spatial-temporal graph representation learning is developed, where a spatial-temporal graph is divided into several sub-graphs on the temporal dimension with a random sub-graph shuffled, and the network is asked to predict the shuffled one to learn the subtle variances of different actions. An attention mechanism called Inception Attention is proposed to highlight the cubes of interest in the spatial-temporal graph, where the dependencies are captured along the spatial and temporal space simultaneously. Experiments on three large-scale public datasets, NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400 verify the effectiveness of our method.