Learning from Machine Learning: advancing from static to dynamic facial function quantification

Objectives We assess an open-source Python machine learning algorithm’s efficacy in image and video analysis of facial palsy (FP) patients. Methods Images and videos of 60 patients with varying FP severities performing standard movements were obtained from MEEI Facial Palsy database. Landmarks generated on images by the open-source algorithm (adapted from OpenCV and Dlib libraries) and Emotrics (standard for two-dimensional FP analysis) were compared. Considering the human eye as the standard for accuracy, three raters marked perceived errors in each algorithm’s tracking of five facial features. Cumulative error distributions between both algorithms were compared via normalized root mean square error. FP severity and facial feature-specific error rates were compared using ANOVA tests. Raters also analyzed open-source algorithm-generated video landmarks; similar statistical comparisons between open-source algorithms’ image and video-based analyses were performed. Results Cumulative error distribution between both algorithms’ image analyses was most robust for normal function; significant discrepancies were observed in mild/moderate flaccidity and nearly-normal/complete synkinesis. Both algorithms had similar error rates across all facial features (p=0.76) and FP severities (p=0.37). In the open-source algorithm’s video analysis, mild synkinesis (24.7%) and severe flaccidity (19.7%) had the highest error rates. Comparing image and video analyses generated by the open-source algorithm, video analyses had lower error rates across all FP severities (p<0.001). Conclusions We report for the first time the feasibility and relative accuracy of a Python open-source algorithm for dynamic facial landmark tracking in FP videos. The demonstrated superiority of landmark tracking with videos over images can improve objective FP quantification.