A Large-Scale Vietnamese Lip Reading Dataset and Enhanced Recognition Pipeline

Visual Speech Recognition (VSR) for Vietnamese has not been extensively explored because of the lack of VSR datasets for the Vietnamese language. In this study, we propose the first Vietnamese Lip Reading (VLR) corpus, namely Vietnamese Lip Reading, consisting of 198 hours of speech and lip data captured from 1,781 videos extracted from online video platforms. The proposed corpus contains word-level alignments with sufficiently high-resolution mouth frames and rich head pose variations with several lighting setups and occlusions. Experiments using six VSR networks based on deep learning approach on VLR achieved an average word error rate of 62.18% when using AV-HuBERT as the base model. To improve on the performance of VSR, a refinement pipeline using a combination of shallow fusion of a Vietnamese 5-gram language model with a difficulty filtering mechanism via curriculum learning, region-of-interest (ROI) detection for lip alignment, and re-ranking using N-best with a lightweight Transformer-based decoder is proposed. Results showed that the proposed refinements were able to bring down the average word error rate to about 30%. The proposed Vietnamese Lip Reading corpus serves as a VSR benchmark dataset enables a systematic exploration of several aspects of VSR on the Vietnamese language, as well as a starting point to develop VSR for other low-resource and tonal languages.