Facial expression recognition plays a significant part in artificial intelligence and computer vision. However, most of facial expression recognition methods have not obtained satisfactory results based on low-level features. The existed methods used in facial expression recognition encountered the major issues of linear inseparability, large computational burden, and data redundancy. To obtain satisfactory results, we propose an innovative deep learning model using the kernel entropy component analysis network and directed acyclic graph (...) support vector machine. We use the KECANet in the feature extraction stage. In the stage of output, binary hashing and blockwise histograms are adopted. We sent the final output features to the DAGSVM classifier for expression recognition. We test the performance of our proposed method on three databases of CK+, JAFFE, and CMU Multi-PIE. According to the experiment results, the proposed method can learn high-level features and provide more recognition information in the stage of training, obtaining a higher recognition rate. (shrink)

Emotional facial expressions can inform researchers about an individual's emotional state. Recent technological advances open up new avenues to automatic Facial Expression Recognition. Based on machine learning, such technology can tremendously increase the amount of processed data. FER is now easily accessible and has been validated for the classification of standardized prototypical facial expressions. However, applicability to more naturalistic facial expressions still remains uncertain. Hence, we test and compare performance of three different FER systems (...) with human emotion recognition for standardized posed facial expressions and for non-standardized acted facial expressions. For the standardized images, all three systems classify basic emotions accurately and they are mostly on par with human raters. For the non-standardized stimuli, performance drops remarkably for all three systems, but Azure still performs similarly to humans. In addition, all systems and humans alike tend to misclassify some of the non-standardized emotional facial expressions as neutral. In sum, emotion recognition by automated facial expression recognition can be an attractive alternative to human emotion recognition for standardized and non-standardized emotional facial expressions. However, we also found limitations in accuracy for specific facial expressions; clearly there is need for thorough empirical evaluation to guide future developments in computer vision of emotional facial expressions. (shrink)

This article discusses how research on the image-based analysis of facial images has informed this debate by demonstrating that a single representational system for facial identity and facial expression is not only computationally viable, but can simulate existing cognitive data demonstrating apparent dissociable processing of these two facial properties. It discusses the increasing number of cognitive studies that provide support for this view. Neuropsychological case studies of brain-injured patients and provide limited evidence for separate visual (...) routes processing facial identity and facial expression. It deals with neuroimaging studies addressing both facial identity and facial expression processing which shows that the fusiform gyrus contributes to the recognition of both facial properties. Research continues to reinforce the proposal that a common visual framework supports the representation of at least some aspects of facial identity and facial expression. (shrink)

This study was designed to test three competing hypotheses (impaired configural processing; impaired Theory of Mind; atypical amygdala functioning) to explain the basic facial expression recognition profile of adults with autism spectrum disorders (ASD). In Experiment 1 the Ekman and Friesen (1976) series were presented upright and inverted. Individuals with ASD were significantly less accurate than controls at recognising upright facial expressions of fear, sadness and disgust and their pattern of errors suggested some configural processing difficulties. (...) Impaired recognition of inverted facial expressions suggested some additional difficulties processing the facial features. Unexpectedly, the clinical group misidentified fear as anger. In Experiment 2 feature processing of facial expressions was investigated by presenting stimuli in a piecemeal fashion, starting with either just the eyes or the mouth. Individuals with ASD were impaired at recognising fear from the eyes and disgust from the mouth; they also confused fearful eyes as being angry. The findings are discussed in terms of the three competing hypotheses tested. (shrink)

There are many ways to assess face perception skills. In this study, we describe a novel task battery FEAST (Facial Expression Action Stimulus Test) developed to test recognition of identity and expressions of human faces as well as stimulus control categories. The FEAST consists of a neutral and emotional face memory task, a face and object identity matching task, a face and house part-to-whole matching task, and a human and animal facial expression matching task. The (...) identity and part-to-whole matching tasks contain both upright and inverted conditions. The results provide reference data of a healthy sample of controls in two age groups for future users of the FEAST. (shrink)

Previous meta-analyses support a female advantage in decoding non-verbal emotion (Hall, 1978, 1984), yet the mechanisms underlying this advantage are not understood. The present study examined whether the female advantage is related to greater female attention to the eyes. Eye-tracking techniques were used to measure attention to the eyes in 19 males and 20 females during a facial expression recognition task. Women were faster and more accurate in their expression recognition compared with men, and women (...) looked more at the eyes than men. Positive relationships were observed between dwell time and number of fixations to the eyes and both accuracy of facial expression recognition and speed of facial expression recognition. These results support the hypothesis that the female advantage in facial expression recognition is related to greater female attention to the eyes. (shrink)

State-of-the-art facial expression methods outperform human beings, especially, thanks to the success of convolutional neural networks. However, most of the existing works focus mainly on analyzing an adult’s face and ignore the important problems: how can we recognize facial expression from a baby’s face image and how difficult is it? In this paper, we first introduce a new face image database, named BabyExp, which contains 12,000 images from babies younger than two years old, and each image (...) is with one of three facial expressions. To the best of our knowledge, the proposed dataset is the first baby face dataset for analyzing a baby’s face image, which is complementary to the existing adult face datasets and can shed some light on exploring baby face analysis. We also propose a feature guided CNN method with a new loss function, called distance loss, to optimize interclass distance. In order to facilitate further research, we provide the benchmark of expression recognition on the BabyExp dataset. Experimental results show that the proposed network achieves the recognition accuracy of 87.90% on BabyExp. (shrink)

This article presents the analysis of the main Spanish political candidates for the elections to be held on April 2019. The analysis focuses on the Facial Expression Analysis, a technique widely used in neuromarketing research. It allows to identify the micro-expressions that are very brief, involuntary. They are signals of hidden emotions that cannot be controlled voluntarily. The video with the final interventions of every candidate has been post-processed using the classification algorithms given by the iMotions's AFFDEX platform. (...) We have then analyzed these data. Firstly, we have identified and compare the basic emotions showed by each politician. Second, we have associated the basic emotions with specific moments of the candidate's speech, identifying the topics they address and relating them directly to the expressed emotion. Third, we have analyzed whether the differences shown by each candidate in every emotion are statistically significant. In this sense, we have applied the non-parametric chi-squared goodness-of-fit test. We have also considered the ANOVA analysis in order to test whether, on average, there are differences between the candidates. Finally, we have checked if there is consistency between the results provided by different surveys from the main media in Spain regarding the evaluation of the debate and those obtained in our empirical analysis. A predominance of negative emotions has been observed. Some inconsistencies were found between the emotion expressed in the facial expression and the verbal content of the message. Also, evidences got from statistical analysis confirm that the differences observed between the various candidates with respect to the basic emotions, on average, are statistically significant. In this sense, this article provides a methodological contribution to the analysis of the public figures' communication, which could help politicians to improve the effectiveness of their messages identifying and evaluating the intensity of the expressed emotions. (shrink)

We used multiple methods to examine two questions about emotion and culture: (1) Which facial expressions are recognised cross-culturally; and (2) does the “forced-choice” method lead to spurious findings of universality? Forty participants in the US and 40 in India were shown 14 facial expressions and asked to say what had happened to cause the person to make the face. Analyses of the social situations given and of the affect words spontaneously used showed high levels of recognition (...) for most of the expressions. A subsequent forced-choice task using the same faces confirmed these findings. Analysis of the pattern of magnitude, discreteness, and similarity of responses across cultures and expressions led to the conclusion that there is no neat distinction between cross-culturally recognisable and nonrecognisable expressions. Results are better described as a gradient of recognition. (shrink)

We used multiple methods to examine two questions about emotion and culture: (1) Which facial expressions are recognised cross-culturally; and (2) does the “forced-choice” method lead to spurious findings of universality? Forty participants in the US and 40 in India were shown 14 facial expressions and asked to say what had happened to cause the person to make the face. Analyses of the social situations given and of the affect words spontaneously used showed high levels of recognition (...) for most of the expressions. A subsequent forced-choice task using the same faces confirmed these findings. Analysis of the pattern of magnitude, discreteness, and similarity of responses across cultures and expressions led to the conclusion that there is no neat distinction between cross-culturally recognisable and nonrecognisable expressions. Results are better described as a gradient of recognition. (shrink)

Deficits in recognition of facial expressions of emotions are considered to be an important factor explaining impairments in social functioning and affective reactions of schizophrenic patients. Many studies confirmed such deficits while controversies remained concerning the emotion valence and modality. The aim of the study was to explore the process of recognizing facial expressions of emotion in the group of schizophrenic patients by analyzing the role of emotion valence, modality and gender of the model. Results of the (...) group of 35 patients and 35 matched controls indicate that while schizophrenic patients show general impairment in recognizing facial expressions of both positive and the majority of negative emotions, there are differences in deficits for particular emotions. Expressions also appeared to be more ambiguous for the patients while variables connected with gender were found less significant. (shrink)

Background: This study developed a photo and video database of 4-to-6-year-olds expressing the seven induced and posed universal emotions and a neutral expression. Children participated in photo and video sessions designed to elicit the emotions, and the resulting images were further assessed by independent judges in two rounds. Methods: In the first round, two independent judges, experts in the Facial Action Coding System, firstly analysed 3,668 emotions facial expressions stimuli from 132 children. Both judges reached 100% agreement (...) regarding 1,985 stimuli, which were then selected for a second round of analysis between judges 3 and 4. Results: The result was 1,985 stimuli were produced from 124 participants. A Kappa index of 0.70 and an accuracy of 73% between experts were observed. Lower accuracy was found for emotional expression by 4-year-olds than 6-year-olds. Happiness, disgust and contempt had the highest agreement. After a sub-analysis evaluation of all four judges, 100% agreement was reached for 1,381 stimuli which compound the ChildEFES database with 124 participants and 51% induced photographs. The number of stimuli of each emotion were: 87 for neutrality, 363 for happiness, 170 for disgust, 104 for surprise, 152 for fear, 144 for sadness, 157 for anger 157, and 183 for contempt. Conclusions: The findings show that this photo and video database can facilitate research on the mechanisms involved in early childhood recognition of facial emotions in children, contributing to the understanding of facial emotion recognition deficits which characterise several neurodevelopmental and psychiatric disorders. (shrink)

ABSTRACTRecognising identity and emotion conveyed by the face is important for successful social interactions and has thus been the focus of considerable research. Debate has surrounded the extent...

Prior studies provide consistent evidence of deficits for psychopaths in processing verbal emotional material but are inconsistent regarding nonverbal emotional material. To examine whether psychopaths exhibit general versus specific deficits in nonverbal emotional processing, 34 psychopaths and 33 nonpsychopaths identified with Hare's (R. D. Hare, 1991) Psychopathy Checklist-Revised were asked to complete a facial affect recognition test. Slides of prototypic facial expressions were presented. Three hypotheses regarding hemispheric lateralization anomalies in psychopaths were also tested (right-hemisphere dysfunction, reduced (...) lateralization, and reversed lateralization). Psychopaths were less accurate than nonpsychopaths at classifying facial affect under conditions promoting reliance on right-hemisphere resources and displayed a specific deficit in classifying disgust. These findings demonstrate that psychopaths exhibit specific deficits in nonverbal emotional processing. (shrink)

Research on facial emotion expression has mostly focused on emotion recognition, assuming that a small number of discrete emotions is elicited and expressed via prototypical facial muscle configurations as captured in still photographs. These are expected to be recognized by observers, presumably via template matching. In contrast, appraisal theories of emotion propose a more dynamic approach, suggesting that specific elements of facial expressions are directly produced by the result of certain appraisals and predicting the (...) class='Hi'>facial patterns to be expected for certain appraisal configurations. This approach has recently been extended to emotion perception, claiming that observers first infer individual appraisals and only then make categorical emotion judgments based on the estimated appraisal patterns, using semantic inference rules. Here we report two related studies to empirically investigate the facial action unit configurations that are used by actors to convey specific emotions in short affect bursts and to examine to what extent observers can infer a person’s emotions from the predicted facial expression configurations. The results show that 1) professional actors use many of the predicted facial action unit patterns to enact systematically specified appraisal outcomes in a realistic scenario setting, and 2) naïve observers infer the respective emotions based on highly similar facial movement configurations with a degree of accuracy comparable to earlier research findings. Based on estimates of underlying appraisal criteria for the different emotions we conclude that the patterns of facial action units identified in this research correspond largely to prior predictions and encourage further research on appraisal-driven expression and inference. (shrink)

Background: Clinical and experimental findings have suggested that insomnia is associated with altered emotion processing, such as facial emotion recognition and impairments in executive functions. However, the results still appear non-consensual and have recently been presented by a few number of studies. Accordingly, the aim of the present study was to investigate whether patients with Insomnia disorder will present alterations in recognition of facial emotions and that such alterations will be related to Executive Functions and that (...) Insomnia Disorder patients will exhibit a lower Executive Functions performance when compared to normal controls individuals. Method: A total of 11 individuals diagnosed with insomnia disorder and 15 control participants took part in the present cross-sectional study. They completed questionnaires covering socio-demographic data, a sleep and mood evaluation, and a cognitive assessment. Moreover, they underwent a computerized Facial Emotional Recognition test, consisting of facial emotion labeling and facial emotion matching. Results: Individuals with insomnia presented a lower recognition of facial expressions of fear and sadness in comparison to control volunteers. Perceptual Organization Index was found to be statistically significant lower in the insomnia group compared to the control group. There was a statistically significant association between facial recognition emotional and cognitive processes of in executive functions performance, especially in processes related to the inhibitory control, planning capacity, problem solving and cognitive flexibility in insomnia group. Conclusions: Insomnia disorder impairs the facial emotion recognition of the fear and sadness. Insomniacs with impairment in the facial emotion recognition had impaired performance of executive functions. There is a negative impact on processes related to cognitive-emotional capacity than on processes related to social-emotional recognition. Keywords: insomnia, facial emotional recognition, sleep deprivation, cognition, executive functions. (shrink)

Automatic facial expression recognition has now advanced to the point that we are able to apply it to spontaneous expressions. The automated tools enable new research activity in cognitive neuroscience, psychiatry, education, human–machine communication, and human social dynamics. Moreover, automated facial expression analysis enables investigations into facial expression dynamics that were previously intractable by human coding because of the time required to code intensity changes. This article provides an overview on the state of (...) the art in computer vision approaches to facial expression recognition, including methods for characterizing expression dynamics. It then discusses behavioral studies that have employed automatic facial expression recognition to learn new information about the relationships of facial expression to internal state, and reviews the first generation of applications in learning and education that take advantage of the real-time expression signal. (shrink)

Following Yik and Russell (1999) a judgement paradigm was used to examine to what extent differential accuracy of recognition of facial expressions allows evaluation of the well-foundedness of different theoretical views on emotional expression. Observers judged photos showing facial expressions of seven emotions on the basis of: (1) discrete emotion categories; (2) social message types; (3) appraisal results; or (4) action tendencies, and rated their confidence in making choices. Emotion categories and appraisals were judged significantly more (...) accurately and confidently than messages or action tendencies. These results do not support claims of primacy for message or action tendency views of facial expression. Based on a componential model of emotion it is suggested that judges can infer components from categories and vice versa. (shrink)

Facial expressions are a vital way for humans to show their perceived emotions. It is convenient for detecting and recognizing expressions or micro-expressions by annotating a lot of data in deep learning. However, the study of video-based expressions or micro-expressions requires that coders have professional knowledge and be familiar with action unit coding, leading to considerable difficulties. This paper aims to alleviate this situation. We deconstruct facial muscle movements from the motor cortex and systematically sort out the relationship (...) among facial muscles, AU, and emotion to make more people understand coding from the basic principles: We derived the relationship between AU and emotion based on a data-driven analysis of 5,000 images from the RAF-AU database, along with the experience of professional coders.We discussed the complex facial motor cortical network system that generates facial movement properties, detailing the facial nucleus and the motor system associated with facial expressions.The supporting physiological theory for AU labeling of emotions is obtained by adding facial muscle movements patterns.We present the detailed process of emotion labeling and the detection and recognition of AU.Based on the above research, the video's coding of spontaneous expressions and micro-expressions is concluded and prospected. (shrink)

ABSTRACTThe recognition of emotional facial expressions is often subject to contextual influence, particularly when the face and the context convey similar emotions. We investigated whether spontaneous, incidental affective theory of mind inferences made while reading vignettes describing social situations would produce context effects on the identification of same-valenced emotions as well as differently-valenced emotions conveyed by subsequently presented faces. Crucially, we found an effect of context on reaction times in both experiments while, in line with previous work, we (...) found evidence for a context effect on accuracy only in Experiment 1. This demonstrates that affective theory of mind inferences made at the pragmatic level of a text can automatically, contextually influence the perceptual processing of emotional facial expressions in a separate task even when those emotions are of a distinctive valence. Thus, our novel findings suggest that language acts as a contex... (shrink)

We discuss here one of our projects, aimed at developing an automatic facial expression interpreter, mainly in terms of signaled emotions. We present some of the relevant findings on facial expressions from cognitive science and psychology that can be understood by and be useful to researchers in Human-Computer Interaction and Artificial Intelligence. We then give an overview of HCI applications involving automated facial expression recognition, we survey some of the latest progresses in this area (...) reached by various approaches in computer vision, and we describe the design of our facial expression recognizer. We also give some background knowledge about our motivation for understanding facial expressions and we propose an architecture for a multimodal intelligent interface capable of recognizing and adapting to computer users¿ affective states. Finally, we discuss current interdisciplinary issues and research questions which will need to be addressed for further progress to be made in the promising area of computational facial expression recognition. (shrink)

Micro-expression is an extremely quick and uncontrollable facial movement that lasts for 40–200 ms and reveals thoughts and feelings that an individual attempts to cover up. Though much more difficult to detect and recognize, ME recognition is similar to macro-expression recognition in that it is influenced by facial features. Previous studies suggested that facial attractiveness could influence facial expression recognition processing. However, it remains unclear whether facial attractiveness could also (...) influence ME recognition. Addressing this issue, this study tested 38 participants with two ME recognition tasks in a static condition or dynamically. Three different MEs at two attractiveness levels. The results showed that participants recognized MEs on attractive faces much quicker than on unattractive ones, and there was a significant interaction between ME and facial attractiveness. Furthermore, attractive happy faces were recognized faster in both the static and the dynamic conditions, highlighting the happiness superiority effect. Therefore, our results provided the first evidence that facial attractiveness could influence ME recognition in a static condition or dynamically. (shrink)

ABSTRACTThere is substantial evidence to suggest that deafness is associated with delays in emotion understanding, which has been attributed to delays in language acquisition and opportunities to converse. However, studies addressing the ability to recognise facial expressions of emotion have produced equivocal findings. The two experiments presented here attempt to clarify emotion recognition in deaf children by considering two aspects: the role of motion and the role of intensity in deaf children’s emotion recognition. In Study 1, 26 (...) deaf children were compared to 26 age-matched hearing controls on a computerised facial emotion recognition task involving static and dynamic expressions of 6 emotions. Eighteen of the deaf and 18 age-matched hearing controls additionally took part in Study 2, involving the presentation of the same 6 emotions at varying intensities. Study 1 showed that deaf children’s emotion recognition was better in the dynamic rather than static condition, whereas the hearing children sh... (shrink)

Facial expressions have communicative properties that bear some importance to perceivers. Such expressions are informative with respect to the future behavior of the expressing individual and with respect to the conditions of the broader social environment. This article argues that appropriate responses to facial expressions are an important means by which people adapt to their social ecology. The immediate responses to facial expressions depend on contextual factors. It is more important for individuals to adapt to the ingroup (...) than to other groups, for this reason people should exhibit special sensitivity to ingroup facial expressions. It reviews the literature regarding the role of context in the recognition of facial expressions and regarding group membership and emotion recognition, with a special emphasis on the role of culture. It focuses on facial expressions of emotion if only because of available empirical literature. (shrink)

The accurate recognition of emotion is important for interpersonal interaction and when navigating our social world. However not all facial displays reflect the emotional experience currently being felt by the expresser. Indeed faces express both genuine and posed displays of emotion. In this article, we summarise the importance of motion for the recognition of face identity before critically outlining the role of dynamic information in determining facial expressions and distinguishing between genuine and posed expressions of emotion. (...) We propose that both dynamic information and face familiarity may modulate our ability to determine whether an expression is genuine or not. Finally, we consider the shared role for dynamic information across different face recognition tasks and the wider impact of face familiarity on determining genuine from posed expressions during real-world interactions. (shrink)

Faces are highly salient objects for humans, providing identity- and emotion-related information—basic cues for mastering social interactions. Despite extensive research on the recognition of facia...