Emotions are multimodal processes that play a crucial role in our everyday lives. Recognizing emotions is becoming more critical in a wide range of application domains such as healthcare, education, human-computer interaction, Virtual Reality, intelligent agents, entertainment, and more. Facial macro-expressions or intense facial expressions are the most common modalities in recognizing emotional states. However, since facial expressions can be voluntarily controlled, they may not accurately represent emotional states. Earlier studies have shown that (...) class='Hi'>facial micro-expressions are more reliable than facial macro-expressions for revealing emotions. They are subtle, involuntary movements responding to external stimuli that cannot be controlled. This paper proposes using facial micro-expressions combined with brain and physiological signals to more reliably detect underlying emotions. We describe our models for measuring arousal and valence levels from a combination of facial micro-expressions, Electroencephalography (EEG) signals, galvanic skin responses (GSR), and Photoplethysmography (PPG) signals. We then evaluate our model using the DEAP dataset and our own dataset based on a subject-independent approach. Lastly, we discuss our results, the limitations of our work, and how these limitations could be overcome. We also discuss future directions for using facial micro-expressions and physiological signals in emotion recognition. (shrink)

Over the last few years, automatic facial micro-expression analysis has garnered increasing attention from experts across different disciplines because of its potential applications in various fields such as clinical diagnosis, forensic investigation and security systems. Advances in computer algorithms and video acquisition technology have rendered machine analysis of facial micro-expressions possible today. Although the study of facial micro-expressions is a well-established field in psychology, it is still relatively new from the computational perspective (...) with many interesting problems. In this survey, we present a comprehensive review of state-of-the-art databases and methods for micro-expressions spotting and recognition. Individual stages involved in the automation of these tasks are also described and reviewed at length. In addition, we also deliberate on the challenges and future directions in this growing field of automatic facial micro-expression analysis. (shrink)

Facial expressions are a vital way for humans to show their perceived emotions. It is convenient for detecting and recognizing expressions or micro-expressions by annotating a lot of data in deep learning. However, the study of video-based expressions or micro-expressions requires that coders have professional knowledge and be familiar with action unit coding, leading to considerable difficulties. This paper aims to alleviate this situation. We deconstruct facial muscle movements from the motor (...) cortex and systematically sort out the relationship among facial muscles, AU, and emotion to make more people understand coding from the basic principles: We derived the relationship between AU and emotion based on a data-driven analysis of 5,000 images from the RAF-AU database, along with the experience of professional coders.We discussed the complex facial motor cortical network system that generates facial movement properties, detailing the facial nucleus and the motor system associated with facial expressions.The supporting physiological theory for AU labeling of emotions is obtained by adding facial muscle movements patterns.We present the detailed process of emotion labeling and the detection and recognition of AU.Based on the above research, the video's coding of spontaneous expressions and micro-expressions is concluded and prospected. (shrink)

Micro-expressions, as fleeting facial expressions, are very important for judging people’s true emotions, thus can provide an essential behavioral clue for lie and dangerous demeanor detection. From embodied accounts of cognition, we derived a novel hypothesis that facial feedback from upper and lower facial regions has differential effects on micro-expression recognition. This hypothesis was tested and supported across three studies. Specifically, the results of Study 1 showed that people became better judges of intense (...) micro-expressions with a duration of 450 ms when the facial feedback from upper face was enhanced via a restricting gel. Additional results of Study 2 showed that the recognition accuracy of subtle micro-expressions was significantly impaired under all duration conditions (50 ms, 150 ms, 333 ms, and 450 ms) when facial feedback from lower face was enhanced. In addition, the results of Study 3 also revealed that blocking the facial feedback of lower face, significantly boosted the recognition accuracy of subtle and intense micro-expressions under all duration conditions (150 ms and 450 ms). Together, these results highlight the role of facial feedback in judging the subtle movements of micro-expressions. (shrink)

Micro-expression is an extremely quick and uncontrollable facial movement that lasts for 40–200 ms and reveals thoughts and feelings that an individual attempts to cover up. Though much more difficult to detect and recognize, ME recognition is similar to macro-expression recognition in that it is influenced by facial features. Previous studies suggested that facial attractiveness could influence facial expression recognition processing. However, it remains unclear whether facial attractiveness could also influence ME recognition. Addressing this (...) issue, this study tested 38 participants with two ME recognition tasks in a static condition or dynamically. Three different MEs at two attractiveness levels. The results showed that participants recognized MEs on attractive faces much quicker than on unattractive ones, and there was a significant interaction between ME and facial attractiveness. Furthermore, attractive happy faces were recognized faster in both the static and the dynamic conditions, highlighting the happiness superiority effect. Therefore, our results provided the first evidence that facial attractiveness could influence ME recognition in a static condition or dynamically. (shrink)

This article presents the analysis of the main Spanish political candidates for the elections to be held on April 2019. The analysis focuses on the Facial Expression Analysis, a technique widely used in neuromarketing research. It allows to identify the micro-expressions that are very brief, involuntary. They are signals of hidden emotions that cannot be controlled voluntarily. The video with the final interventions of every candidate has been post-processed using the classification algorithms given by the iMotions's AFFDEX (...) platform. We have then analyzed these data. Firstly, we have identified and compare the basic emotions showed by each politician. Second, we have associated the basic emotions with specific moments of the candidate's speech, identifying the topics they address and relating them directly to the expressed emotion. Third, we have analyzed whether the differences shown by each candidate in every emotion are statistically significant. In this sense, we have applied the non-parametric chi-squared goodness-of-fit test. We have also considered the ANOVA analysis in order to test whether, on average, there are differences between the candidates. Finally, we have checked if there is consistency between the results provided by different surveys from the main media in Spain regarding the evaluation of the debate and those obtained in our empirical analysis. A predominance of negative emotions has been observed. Some inconsistencies were found between the emotion expressed in the facial expression and the verbal content of the message. Also, evidences got from statistical analysis confirm that the differences observed between the various candidates with respect to the basic emotions, on average, are statistically significant. In this sense, this article provides a methodological contribution to the analysis of the public figures' communication, which could help politicians to improve the effectiveness of their messages identifying and evaluating the intensity of the expressed emotions. (shrink)

This article analyzes the role of emotions in social interaction and their effects on social structuration and the emergence of micro-social order. It argues that facial expressions of emotion are key in generating robust patterns of social interaction. First, the article shows that actors' encoding of facial expressions combines hardwired physiological principles on the one hand and socially learned aspects on the other hand, leading to fine-grained and socially differentiated dialects of expression. Second, it is (...) argued that decoding facial expression is contingent upon this combination so that reciprocal attributions of emotional states, situational interpretations, and action tendencies are more effective within rather than across social units. Third, this conjunction affects the conditions for emotional contagion, which is argued to be more effective within social units exhibiting similar encoding and decoding characteristics, and thus aligns emotions and action tendencies in a coherent, yet socially differentiated way. (shrink)

Resumen El reconocimiento facial es una tecnología que se ha utilizado para el control de la población a fin de combatir los efectos de la pandemia por COVID-19. Sin embargo, esta tecnología se utiliza hace bastante tiempo en muchos ámbitos del quehacer económico, social y político. Este trabajo analiza el reconocimiento fa cial entendida como la captura de micro expresiones faciales mediante la cual se puede medir y evaluar emociones con el fin de vigilar movimiento, predecir com portamientos (...) y anticipar intenciones de compra y de voto. Nuestro objetivo es revisar los enfoques teóricos que sustentan esta tecnología que ha devenido en una nueva oportunidad de negocio, para proponer después algunas reflexiones éticas derivadas.Facial recognition is a technology that is used to control people in order to fight the impacts of COVID-19. However, this technology has been used formany years in many economical, social and political areas. This work provides an analysis of the new business based on technologies of facial micro-expressions recognition. By means of this procedure, the involved companies measure and evaluate emotions with the aim of monitoring the movement, predicting behaviors, intentions of purchase and vote, between other applications. Our purpose is to check the theoretical suppositions that sustain this business, to propose later some ethical derivative reflections. (shrink)

Past research has found that skin colouration, particularly facial redness, influences the perceived health and emotional state of target individuals. In the current work, we explore several extensions of this past research. In Experiment 1, we manipulated facial redness incrementally on neutral and angry faces and had participants rate each face for anger and health. Different red effects emerged, as perceived anger increased in a linear manner as facial redness increased. Health ratings instead showed a curvilinear trend, (...) as both extreme paleness and redness were rated as less healthy than moderate levels of red. Experiment 2 replicated and extended these findings by manipulating the masculinity of both angry and neutral faces that varied in redness. The results found the effect of red on perceived anger and health was moderated by masculine face structure. Collectively, these results show that facial redness has context dependent effects that vary based on facial expression, appearance, and differentially impact ratings of emotional states and health. (shrink)

Accurate perception of the emotional signals conveyed by others is crucial for successful social interaction. Such perception is influenced not only by sensory input, but also by knowledge we have about the others’ emotions. This study addresses the issue of whether knowing that the other’s emotional state is congruent or incongruent with their displayed emotional expression (“genuine” and “fake”, respectively) affects the neural mechanisms underpinning the perception of their facial emotional expressions. We used a visual adaptation paradigm to (...) investigate this question in three experiments employing increasing adaptation durations. The adapting stimuli consisted of photographs of emotional facial expressions of joy and anger, purported to reflect (in-)congruency between felt and expressed emotion, displayed by professional actors. A Validity checking procedure ensured participants had the correct knowledge about the (in-)congruency. Significantly smaller adaptation aftereffects were obtained when participants knew that the displayed expression was incongruent with the felt emotion, following all tested adaptation periods. This study shows that knowledge relating to the congruency between felt and expressed emotion modulates face expression aftereffects. We argue that this reflects that the neural substrate responsible for the perception of facial expressions of emotion incorporates the presumed felt emotion underpinning the expression. (shrink)

Micro-expressions can reflect an individual’s subjective emotions and true mental state, and they are widely used in the fields of mental health, justice, law enforcement, intelligence, and security. However, one of the major challenges of working with MEs is that their neural mechanism is not entirely understood. To the best of our knowledge, the present study is the first to use electroencephalography to investigate the reorganizations of functional brain networks involved in MEs. We aimed to reveal the underlying (...) neural mechanisms that can provide electrophysiological indicators for ME recognition. A real-time supervision and emotional expression suppression experimental paradigm was designed to collect video and EEG data of MEs and no expressions of 70 participants expressing positive emotions. Based on the graph theory, we analyzed the efficiency of functional brain network at the scalp level on both macro and micro scales. The results revealed that in the presence of MEs compared with NEs, the participants exhibited higher global efficiency and nodal efficiency in the frontal, occipital, and temporal regions. Additionally, using the random forest algorithm to select a subset of functional connectivity features as input, the support vector machine classifier achieved a classification accuracy for MEs and NEs of 0.81, with an area under the curve of 0.85. This finding demonstrates the possibility of using EEG to recognize MEs, with a wide range of application scenarios, such as persons wearing face masks or patients with expression disorders. (shrink)

Women typically report more pain than men, as well as exhibit specific sex differences in the perception and emotional expression of pain. We present evidence that sex is a significant variable in the evolution of facial expression of pain.

Research on facial emotion expression has mostly focused on emotion recognition, assuming that a small number of discrete emotions is elicited and expressed via prototypical facial muscle configurations as captured in still photographs. These are expected to be recognized by observers, presumably via template matching. In contrast, appraisal theories of emotion propose a more dynamic approach, suggesting that specific elements of facial expressions are directly produced by the result of certain appraisals and predicting the facial (...) patterns to be expected for certain appraisal configurations. This approach has recently been extended to emotion perception, claiming that observers first infer individual appraisals and only then make categorical emotion judgments based on the estimated appraisal patterns, using semantic inference rules. Here we report two related studies to empirically investigate the facial action unit configurations that are used by actors to convey specific emotions in short affect bursts and to examine to what extent observers can infer a person’s emotions from the predicted facial expression configurations. The results show that 1) professional actors use many of the predicted facial action unit patterns to enact systematically specified appraisal outcomes in a realistic scenario setting, and 2) naïve observers infer the respective emotions based on highly similar facial movement configurations with a degree of accuracy comparable to earlier research findings. Based on estimates of underlying appraisal criteria for the different emotions we conclude that the patterns of facial action units identified in this research correspond largely to prior predictions and encourage further research on appraisal-driven expression and inference. (shrink)

Many nonhuman animals produce facial expressions which sometimes bear clear resemblance to the facial expressions seen in humans. An understanding of this evolutionary continuity between species, and how this relates to social and ecological variables, can help elucidate the meaning, function, and evolution of facial expression. This aim, however, requires researchers to overcome the theoretical and methodological differences in how human and nonhuman facial expressions are approached. Here, we review the literature relating to (...) nonhuman facial expressions and suggest future directions that could facilitate a better understanding of facial expression within an evolutionary context. (shrink)

This paper proposes that human expression of pain in the presence or absence of caregivers, and the detection of pain by observers, arises from evolved propensities. The function of pain is to demand attention and prioritise escape, recovery, and healing; where others can help achieve these goals, effective communication of pain is required. Evidence is reviewed of a distinct and specific facial expression of pain from infancy to old age, consistent across stimuli, and recognizable as pain by observers. Voluntary (...) control over amplitude is incomplete, and observers can better detect pain that the individual attempts to suppress rather than amplify or simulate. In many clinical and experimental settings, the facial expression of pain is incorporated with verbal and nonverbal vocal activity, posture, and movement in an overall category of pain behaviour. This is assumed by clinicians to be under operant control of social contingencies such as sympathy, caregiving, and practical help; thus, strong facial expression is presumed to constitute an attempt to manipulate these contingencies by amplification of the normal expression. Operant formulations support skepticism about the presence or extent of pain, judgments of malingering, and sometimes the withholding of caregiving and help. To the extent that pain expression is influenced by environmental contingencies, however, could equally plausibly constitute the release of suppression according to evolved contingent propensities that guide behaviour. Pain has been largely neglected in the evolutionary literature and the literature on expression of emotion, but an evolutionary account can generate improved assessment of pain and reactions to it. (shrink)

That facial expressions are universal emotion signals has been supported by observers agreeing on the emotion mimed by actors. We show that actors can mime a diverse range of states: emotions, cognitions, physical states, and actions. English, Hindi, and Malayalam speakers viewed 25 video clips and indicated the state conveyed. Within each language, at least 23 of the 25 clips were recognised above chance and base rate. Facial expressions of emotions are not special in their recognisability, (...) and it is miming that may be the universal human ability. (shrink)

This study was designed to test three competing hypotheses (impaired configural processing; impaired Theory of Mind; atypical amygdala functioning) to explain the basic facial expression recognition profile of adults with autism spectrum disorders (ASD). In Experiment 1 the Ekman and Friesen (1976) series were presented upright and inverted. Individuals with ASD were significantly less accurate than controls at recognising upright facial expressions of fear, sadness and disgust and their pattern of errors suggested some configural processing difficulties. Impaired (...) recognition of inverted facial expressions suggested some additional difficulties processing the facial features. Unexpectedly, the clinical group misidentified fear as anger. In Experiment 2 feature processing of facial expressions was investigated by presenting stimuli in a piecemeal fashion, starting with either just the eyes or the mouth. Individuals with ASD were impaired at recognising fear from the eyes and disgust from the mouth; they also confused fearful eyes as being angry. The findings are discussed in terms of the three competing hypotheses tested. (shrink)

In honoring Carroll Izard’s contributions to emotion research, we discuss infant facial activity and emotion expression. We consider the debated issue of whether infants are biologically prepared to express specific emotions. We offer a perspective that potentially integrates differing viewpoints on infant facial expression of emotion. Specifically, we suggest that evolution has prepared infants with innate action readiness patterns, which are crucial for early infant–caregiver social interaction, and in the course of social interaction specific facial configurations acquire (...) functional significance, becoming associated with specific emotions. Research has not confirmed the presence of innate neurophysiological action patterns that map onto discrete emotions but evidence indicates that the possibility has not been ruled out. (shrink)

This article aims to determine to what extent photographic practices in psychology, psychiatry and physiology contributed to the definition of the external bodily signs of passions and emotions in the second half of the 19th century in France. Bridging the gap between recent research in the history of emotions and photographic history, the following analyses focus on the photographic production of scientists and photographers who made significant contributions to the study of expressions and gestures, namely Duchenne de Boulogne, Charles (...) Darwin, Paul Richer and Albert Londe. This article argues that photography became a key technology in their works due to the adequateness of the exposure time of different cameras to the duration of the bodily manifestations to be recorded, and that these uses constituted facial expressions and bodily gestures as particular objects for the scientific study. (shrink)

Emotional development is one of the largest and most productive areas of psychological research. For decades, researchers have been fascinated by how humans respond to, detect, and interpret emotional facial expressions. Much of the research in this area has relied on controlled stimulus sets of adults posing various facial expressions. Here we introduce a new stimulus set of emotional facial expressions into the domain of research on emotional development—The Child Affective Facial Expression set (...) (CAFE). The CAFE set features photographs of a racially and ethnically diverse group of 2- to 8-year-old children posing for six emotional facial expressions—angry, fearful, sad, happy, surprised, and disgusted—and a neutral face. In the current work, we describe the set and report validity and reliability data on the set from 100 untrained adult participants. (shrink)

Following Yik and Russell (1999) a judgement paradigm was used to examine to what extent differential accuracy of recognition of facial expressions allows evaluation of the well-foundedness of different theoretical views on emotional expression. Observers judged photos showing facial expressions of seven emotions on the basis of: (1) discrete emotion categories; (2) social message types; (3) appraisal results; or (4) action tendencies, and rated their confidence in making choices. Emotion categories and appraisals were judged significantly more (...) accurately and confidently than messages or action tendencies. These results do not support claims of primacy for message or action tendency views of facial expression. Based on a componential model of emotion it is suggested that judges can infer components from categories and vice versa. (shrink)

Facial expressions of pain may be best conceptualized as an example of an evolved propensity to communicate distress, rather than as a distinct category of facial expression. The operant model goes beyond the evolutionary account, as it can explain how the (facial) expression of pain can become maladaptive as a result of its capability to elicit attention and caring behavior in the observer.

In this article, we review empirical evidence regarding the relationship between facial expression and emotion during infancy. We focus on differential emotions theory’s view of this relationship because of its theoretical and methodological prominence. We conclude that current evidence fails to support its proposal regarding a set of pre-specified facial expressions that invariably reflect a corresponding set of discrete emotions in infants. Instead, the relationship between facial expression and emotion appears to be more complex. Some (...) class='Hi'>facial expressions may have different meanings in infants than in children and adults. In addition, nonemotion factors may sometimes lead to the production of “emotional” facial expressions. We consider alternative perspectives on the nature of emotion and emotional expression in infancy with particular focus on differentiation and dynamical systems approaches. (shrink)

This article discusses how research on the image-based analysis of facial images has informed this debate by demonstrating that a single representational system for facial identity and facial expression is not only computationally viable, but can simulate existing cognitive data demonstrating apparent dissociable processing of these two facial properties. It discusses the increasing number of cognitive studies that provide support for this view. Neuropsychological case studies of brain-injured patients and provide limited evidence for separate visual routes (...) processing facial identity and facial expression. It deals with neuroimaging studies addressing both facial identity and facial expression processing which shows that the fusiform gyrus contributes to the recognition of both facial properties. Research continues to reinforce the proposal that a common visual framework supports the representation of at least some aspects of facial identity and facial expression. (shrink)

This paper proposes that human expression of pain in the presence or absence of caregivers, and the detection of pain by observers, arises from evolved propensities. The function of pain is to demand attention and prioritise escape, recovery, and healing; where others can help achieve these goals, effective communication of pain is required. Evidence is reviewed of a distinct and specific facial expression of pain from infancy to old age, consistent across stimuli, and recognizable as pain by observers. Voluntary (...) control over amplitude is incomplete, and observers can better detect pain that the individual attempts to suppress rather than amplify or simulate. In many clinical and experimental settings, the facial expression of pain is incorporated with verbal and nonverbal vocal activity, posture, and movement in an overall category of pain behaviour. This is assumed by clinicians to be under operant control of social contingencies such as sympathy, caregiving, and practical help; thus, strong facial expression is presumed to constitute an attempt to manipulate these contingencies by amplification of the normal expression. Operant formulations support skepticism about the presence or extent of pain, judgments of malingering, and sometimes the withholding of caregiving and help. To the extent that pain expression is influenced by environmental contingencies, however, “amplification” could equally plausibly constitute the release of suppression according to evolved contingent propensities that guide behaviour. Pain has been largely neglected in the evolutionary literature and the literature on expression of emotion, but an evolutionary account can generate improved assessment of pain and reactions to it. Key Words: adaptation; evolutionary psychology; facial expression; pain. (shrink)

The experience of pain appears to be associated, from early infancy and across pain stimuli, with a consistent facial expression in humans. A social function is proposed for this: the communication of pain and the need for help to observers, to whom information about danger is of value, and who may provide help within a kin or cooperative relationship. Some commentators have asserted that the evidence is insufficient to account for the consistency of the face, as judged by technical (...) means or in the perceptions of observers, or that facial expression is epiphenomenal to a gross behavioural defensive response to pain. The major criticism is that it is unnecessary to invoke evolutionary mechanisms beyond the emergence of an unconditioned facial response to pain in neonates, subsequently shaped and maintained by instrumental and social reinforcement throughout life. These criticisms are addressed, acknowledging the need for further data to address some, and elaborating the areas in which evolutionary and operant mechanisms would predict different behavioural interactions, rather than acting synergistically. Several supportive commentaries propose extending evolutionarily-based hypotheses to sex differences, the complexities of others' responses within the social relationship, and the role of empathy. A number of commentators provided valuable suggestions for experimental paradigms or methodological issues. Overall, addressing these issues indicates the need for further conceptual development and for collection of data specifically in relation to these hypotheses. (shrink)

The experience of pain appears to be associated, from early infancy and across pain stimuli, with a consistent facial expression in humans. A social function is proposed for this: the communication of pain and the need for help to observers, to whom information about danger is of value, and who may provide help within a kin or cooperative relationship. Some commentators have asserted that the evidence is insufficient to account for the consistency of the face, as judged by technical (...) means or in the perceptions of observers, or that facial expression is epiphenomenal to a gross behavioural defensive response to pain. The major criticism is that it is unnecessary to invoke evolutionary mechanisms beyond the emergence of an unconditioned facial response to pain in neonates, subsequently shaped and maintained by instrumental and social reinforcement throughout life. These criticisms are addressed, acknowledging the need for further data to address some, and elaborating the areas in which evolutionary and operant mechanisms would predict different behavioural interactions, rather than acting synergistically. Several supportive commentaries propose extending evolutionarily-based hypotheses to sex differences, the complexities of others' responses within the social relationship, and the role of empathy. A number of commentators provided valuable suggestions for experimental paradigms or methodological issues. Overall, addressing these issues indicates the need for further conceptual development and for collection of data specifically in relation to these hypotheses. (shrink)

Facial expression recognition plays a significant part in artificial intelligence and computer vision. However, most of facial expression recognition methods have not obtained satisfactory results based on low-level features. The existed methods used in facial expression recognition encountered the major issues of linear inseparability, large computational burden, and data redundancy. To obtain satisfactory results, we propose an innovative deep learning model using the kernel entropy component analysis network and directed acyclic graph support vector machine. We use the (...) KECANet in the feature extraction stage. In the stage of output, binary hashing and blockwise histograms are adopted. We sent the final output features to the DAGSVM classifier for expression recognition. We test the performance of our proposed method on three databases of CK+, JAFFE, and CMU Multi-PIE. According to the experiment results, the proposed method can learn high-level features and provide more recognition information in the stage of training, obtaining a higher recognition rate. (shrink)

The proposal that there are specific adaptations for the expression and detection of pain appears premature on both conceptual and empirical grounds. We discuss criteria for the validation of a pain facial expression. We also describe recent findings from our lab on coping styles and pain expression, which illustrate the importance of considering individual differences when proposing evolutionary explanations.