New research shows Artificial Intelligence still lags behind humans when it comes to recognising emotions
New DCU led research into the accuracy of artificial intelligence when it comes to reading emotions on our faces has shown that it still lags behind human observers when it comes to being able to tell whether we’re happy or sad. The difference was particularly pronounced when it came to spontaneous displays of emotion.
The recently published study, A performance comparison of eight commercially available automatic classifiers for facial affect recognition, looked at eight “out of the box” automatic classifiers for facial affect recognition (artificial intelligence that can identify human emotions on faces) and compared their emotion recognition performance to that of human observers.
It found that the human recognition accuracy of emotions was 72% whereas among the artificial intelligence tested, the researchers observed a large variance in recognition accuracy, ranging from 48% to 62%.
The work was conducted by Dr. Damien Dupré from Dublin City University’s Business School, Dr. Eva Krumhuber from the Department of Experimental Psychology at UCL, Dr. Dennis Küster from the Cognitive Systems Lab, University of Bremenand Dr. Gary J. McKeown from the Department of Psychology at Queen’s University Belfast.
Key data points
- Eight out-of-the-box automatic classifiers tested.
- 937 videos were sampled from two large databases that conveyed the basic six emotions (happiness, sadness, anger, fear, surprise, and disgust).
- The study examined both posed and spontaneous emotions.
- Results revealed a significant recognition advantage for human observers over automatic classification (72% for human observers)
- Among the eight classifiers, there was considerable variance in recognition accuracy ranging from 48% to 62%.
- Classification accuracy for AI was consistently lower for spontaneous affective behavior.
- The findings indicate shortcomings of existing out-of-the-box classifiers for measuring emotions.
How the study was done
Two well-known dynamic facial expression databases were chosen: BU-4DFE from Binghamton University in New York and the other from The University of Texas in Dallas.
Both are annotated in terms of emotion categories, and contain either posed or spontaneous facial expressions. All of the examined expressions were dynamic to reflect the realistic nature of human facial behavior.
To evaluate the accuracy of emotion recognition, the study compared the performance achieved by human judges with those of eight commercially available automatic classifiers.
Dr. Damien Dupré said
“AI systems claiming to recognise humans’ emotions from their facial expressions are now very easy to develop. However, most of them are based on inconclusive scientific evidence that people are expressing emotions in the same way.
For these systems, human emotions come down to only six basic emotions, but they do not cope well with blended emotions.
Companies using such systems need to be aware that the results obtained are not a measure of the emotion felt, but merely a measure of how much one’s face matches with a face supposed to correspond to one of these six emotions."
Co-author Dr. Eva Krumhuber from UCL added
“AI has come a long way in identifying people’s facial expressions, but our research suggests that there is still room for improvement in recognising genuine human emotions.”
Dr. Krumhuber recently led a separate study published in Emotion (also involving Dr. Küster) comparing human vs. machine recognition across fourteen different databases of dynamic facial expressions.
Researchers
Dr. Damien Dupré - Business School, Dublin City University
Dr. Eva Krumhuber - Department of Experimental Psychology, UCL
Dr. Dennis Küster - Cognitive Systems Lab, University of Bremen
Dr. Gary J. McKeown - Department of Psychology, Queen’s University Belfast
The full paper can be read on PLOS ONE here.
Photo by Andrea Piacquadio from Pexels