Machine and human emotion classification diverge for naturalistic images

The pursuit of affective computing aims to endow machines with the ability to read and respond to human emotions. Unfortunately, several high-profile publicly available machine classifiers are based on psychological theory that over-emphasises the prototypical, morphological features of six or seven basic emotional expressions. This approach is poorly suited to the complex and nuanced repertoire of naturalistic facial expressions; it also omits other physical information that humans use in emotion perception such as tears, eye-gaze, and facial colouring. Prior studies have focused on lab-generated expression stimuli, which are mostly posed, and evaluated machine and human classification against database labels. Here, we use a large, novel set of naturalistic expression stimuli (2,453 still images from 824 YouTube clips) to directly compare machine (Affdex) and human emotion classification. Overall, we found that human-machine agreement was strikingly low. At best, humans and Affdex agreed on two-thirds of happy stimuli, but this was more than double the rate for the next agreed-upon emotion (surprise = 33%). Agreement was under twelve percent for the remaining five emotion categories, with Affdex detecting only prototypical expressions. The findings underscore the importance of designing machine systems that can adapt to the subtleties of human emotional expressions for greater human-machine consistency.