Learning a sparse codebook of facial and body microexpressions for emotion recognition

Context-aware computing has become an essential component of modern wearable technology, enabling devices to sense, interpret, and respond to users’ environments and affective states. By integrating contextual intelligence, smart systems can adapt their own behavior to better align with user needs, enhancing efficiency, personalization, and user experience. A critical aspect of context awareness is the ability to recognize a user’s internal state, particularly emotions, which influence decision-making, communication, and overall well-being. While emotion recognition is traditionally approached through multimodal sensing—using cameras, microphones, or physiological sensors such as Electroencephalography (EEG), Electrocardiography (ECG), and Electro-dermal Activity (EDA)—these methods often require multiple sensors, leading to challenges related to power consumption, complexity, and privacy. The demand for a more efficient and unobtrusive approach to emotion and context recognition has driven research toward minimal-sensor solutions that can extract meaningful affective and contextual information with reduced hardware requirements. Restricting sensing sources of contextual information by using only limited number of sensors might come at the expense of constraining the ultimate goal of context-awareness. Such expense can be mitigated by “doing more with less” i.e. delving into the opportunities of using a single sensor for multiple purposes. What is desirable is a minimal-sensor solution capable of capturing both motion and physiological signals for both robust emotion recognition and additional context estimation. Among wearable sensors, chest-worn Inertial Measurement Units (IMUs) stand out as promising candidates for comprehensive context estimation, as they are well-positioned to capture motion patterns, and physiological vibrations, being the subtle cardio-respiratory signals. This raises the question of whether a single chest-worn IMU could serve as a minimal yet effective sensor for simultaneously estimating both emotional and contextual information, offering a practical alternative to traditional multi-sensor approaches. To address the challenge of using chest-worn IMU for multiple context sensing, I investigated the potential of using Seismocardiography (SCG) and Accelerometry-Derived Respiration (ADR) to estimate emotional states. First, I explored the existing capabilities of chest-worn IMUs in capturing cardio-respiratory vibrations and motion data, which I used as a knowledge base for embedding such IMUs in context sensing. Data plays a crucial role in this problem, therefore I developed a flexible software interface, ColEmo, to facilitate the scalable collection of emotion-related data and introduced a novel dataset, EmoWear, integrating physiological and motion data for emotion recognition and context awareness. Then, I examined the feasibility of using SCG for emotion recognition, comparing its performance with traditional physiological signals and evaluating its effectiveness as a single-sensor solution. This thesis presents a detailed analysis of the potential of chest-worn IMUs for emotion and context recognition. It reports on the design and validation of a SCG-based wearable system, the development and characteristics of the EmoWear dataset, and the implementation of the ColEmo platform for emotion data collection. The research also examines machine learning techniques applied to the SCG and ADR signals, assessing their effectiveness in estimating emotional states. My findings indicate that SCG-based emotion recognition achieves performance comparable to conventional physiological counterparts, demonstrating the feasibility of using a single IMU sensor for affective computing. The publicly-available EmoWear dataset provides a valuable resource for future research, highlighting the potential of combining motion and physiological data for enhanced context awareness. Additionally, the ColEmo software interface, which I published open-source, proves to be an effective tool for multi-modal emotion data collection, facilitating further advancements in wearable emotion recognition. The results pave the way for a minimal-sensor approach for context information estimation that can significantly reduce system complexity while maintaining high performance. These findings confirm that chest-worn IMUs present an opportunity to further improve emotion recognition in addition to the motion-based context information estimation. My research highlights the need for standardized methodologies in affective computing and demonstrates the viability of motion-based physiological sensing for emotion estimation. By using a single chest-worn IMU, this work presents an alternative to multi-sensor systems, reducing power consumption, and privacy concerns while maintaining robust performance. Future research should focus on refining IMU-based emotion recognition using deep learning techniques and larger datasets to enhance classification accuracy. Further validation in real-world settings is necessary to assess the practical applicability of the proposed system. The EmoWear dataset should be explored for additional context-awareness applications, such as activity recognition and stress monitoring, to maximize its utility. The ColEmo platform offers opportunities for expansion beyond emotion recognition, including cognitive load estimation, stress monitoring, and human-computer interaction. This research moves wearable technology forward by using fewer sensors, helping make affective computing more efficient, versatile, and respectful of privacy.

In advertising, identifying the content safety of web pages is a significant concern since advertisers do not want brands to be associated with threatening content. At the same time, publishers would like to maximize the number of web pages on which they can place ads. Thus, a fine balance must be achieved while classifying content safety in order to satisfy both advertisers and publishers. In this paper, we propose a multimodal machine learning framework that fuses visual and textual information from web pages to improve current predictions of content safety. The primary focus is on late fusion, which involves combining final model outputs of separate modalities, such as images and text, to arrive at a single decision. This paper presents a fully automated machine learning framework that performs binary and multilabel classification using late fusion techniques. We also introduce additional work in early fusion, which involves extracting and fusing intermediate features from the two separate models. Our algorithms are applied to data extracted from relevant web pages in the advertising industry. Both of our late and early fusion methods obtain significant improvements over algorithms currently in use.

Research shows a reduced responsivity to implicit as well as explicit facial emotion recognition (emotional dampening) in prehypertensives and hypertensives. This study explored auditory and audiovisual emotion recognition in prehypertensives and hypertensives. Participants (N = 175) who were normotensives, prehypertensives, and hypertensives (n = 57, 58, and 60, respectively) completed an auditory implicit task (matching auditory target with auditory distractors) and two cross‐modal implicit tasks (matching visual target with auditory distractors, and vice‐versa), and an auditory explicit task (labelling emotions in audio‐clips). Findings showed an aberrant speed–accuracy trade‐off, where prehypertensives focused more on accuracy at the cost of speed while hypertensives showed the opposite. Discriminant function analysis revealed that blood pressure (BP)‐associated emotional dampening is a highly specific but moderately sensitive correlate of hypertension. Our study highlights that prehypertensives and hypertensives demonstrate emotional dampening in implicit (but not explicit) auditory emotion recognition and a greater deficit for auditory than visual recognition of implicit emotions. Findings show emotional dampening as an observable correlate of elevated BP and hypertension.

Emotion recognition has been one of the cornerstones of human-computer interaction. Although decades of work has attacked the problem of automatic emotion recognition from either audio or video signals, the fusion of the two modalities is more recent. In this paper, we aim to tackle the problem when both audio and video data are available in a synchronized manner. We address the six basic human emotions, namely, anger, disgust, fear, happiness, sadness, and surprise. We employ an automatic face tracker to extract the different facial points of interest from a video. We then compute feature vectors for each video frame using distances and angles between the tracked points. For audio data, we use the pitch, energy and MFCC to derive feature vectors for each window as well as the entire audio signal. We use two standard techniques, GMM-based HMM and SVM, as the base classifiers. We then design a novel fusion method using the F-score of the base classifiers. We first demonstrate that our fusion approach can increase the accuracy of the base classifiers by as much as 5%. Finally, we show that our fusion-based bi-modal emotion recognition method achieves an overall accuracy of 54% on a publicly available database, which is an improvement upon the current state-of-the-art by 9%.

In this paper we explore automatic genre labelling of motion picture previews using audio-visual features present in movie trailers and the focus is on fusion techniques (early fusion and late fusion) and the resultant improvement on classification accuracy. This paper proposes a novel combination of deep learned features (from a pretrained VGG-16 model) obtained using a state-of-the-art shot detector and hand-crafted audio features. This combination of features and an associated comparison of early and late fusion with these features has not been attempted in the literature before. Furthermore, two popular fusion techniques and three distinct classification algorithms are investigated to determine the optimal fusion technique and classifier combination. The study uses a subset of the LMTD-9 movie trailer dataset with selected genres (action, comedy, drama and horror). The best performing low-level audio features are comprised of timbre features extracted using the MIRtoolbox followed by standalone mel-frequency cepstral coefficients. The best performing high-level audio feature is tonality. Audio features are augmented by visual features extracted using a pre-trained convolutional neural network (VGG-16). Feature fusion (early and late fusion) methods are investigated together with classification methods such as extreme gradient boosting, support vector machine and a neural network. Evaluation metrics such as precision, recall, confusion matrices and F1 score are used to measure classification accuracy. Early fusion methods outperform late fusion methods with a classification performance gain of approximately 10% for a four class classification problem. The best classification performance for early fusion obtained with a support vector machine is (73.12% accuracy), followed by the extreme gradient boosting classifier (69.37% accuracy) and neural network classifier (67.50% accuracy), whereas chance is 25%. It is shown that superior classification performance can be achieved by employing early feature fusion of low-level audio descriptors, high-level audio descriptors and high-level visual feature descriptors together with suitable classifiers.

Due to the recent increase in non-face-to-face services due to COVID-19, the number of users communicating through messengers or SNS (social networking service) is increasing. As a large amount of data is generated by users, research on recognizing emotions by analyzing user information or opinions is being actively conducted. Conversation data such as SNS is freely created by users, so there is no set format. Due to these characteristics, it is difficult to analyze using AI (artificial intelligence), which leads to a decrease in the performance of the emotion recognition technique. Therefore, a processing method suitable for the characteristics of unstructured data is required. Among the unstructured data, most emotion recognition in Korean conversation recognizes a single emotion by analyzing emotion keywords or vocabulary. However, since multiple emotions exist complexly in a single sentence, research on multilabel emotion recognition is needed. Therefore, in this paper, the characteristics of unstructured conversation data are considered and processed for more accurate emotion recognition. In addition, we propose a multilabel emotion recognition technique that understands the meaning of dialogue and recognizes inherent and complex emotions. A deep learning model was compared and tested as a method to verify the usefulness of the proposed technique. As a result, performance was improved when it was processed in consideration of the characteristics of unstructured conversation data. Also, when the attention model was used, accuracy showed the best performance with 65.9%. The proposed technique can contribute to improving the accuracy and performance of conversational emotion recognition.

In this article the role of different categories of postures in the detection, recognition, and interpretation of emotion in contextually rich scenarios, including ironic items, is investigated. Animated scenarios are designed with 3D virtual agents in order to test 3 conditions: In the “still” condition, the narrative content was accompanied by emotional facial expressions without any body movements; in the “idle” condition, emotionally neutral body movements were introduced; and in the “congruent” condition, emotional body postures congruent with the character's facial expressions were displayed. Those conditions were examined by 27 subjects, and their impact on the viewers’ attentional and emotional processes was assessed. The results highlight the importance of the contextual information to emotion recognition and irony interpretation. It is also shown that both idle and emotional postures improve the detection of emotional expressions. Moreover, emotional postures increase the perceived intensity of emotions and the realism of the animations.

Automatic emotion recognition from speech is limited by the ability to discover the relevant predicting features. The common approach is to extract a very large set of features over a generally long analysis time window. In this paper we investigate the applicability of two-sample Kolmogorov-Smirnov statistical test (KST) to the problem of segmental speech emotion recognition. We train emotion classifiers for each speech segment within an utterance. The segment labels are then combined to predict the dominant emotion label. Our findings show that KST can be successfully used to extract statistically relevant features. KST criterion is used to optimize the parameters of the statistical segmental analysis, namely the window segment size and shift. We carry out seven binary class emotion classification experiments on the Emo-DB and evaluate the impact of the segmental analysis and emotion-specific feature selection.

Speech Emotion Recognition (SER) is a challenge task due to the complexity and bias of emotion and environment. Unlike facial emotion recognition or text emotion recognition, speech signal has its unique characteristics that are hard to be recognized by machine. Sometimes even human cannot recognize it due to personal bias. With help of some professionally annotated voice databases, some researches use Mel-Spectrogram as input and attention-mechanism-based Convolutional and Recurrent Neural Network (ARCNN) as training approach. The ARCNN method significantly outperforms Support Vector Machine (SVM) method on public databases. However, few of them apply the techniques on real data. We choose customer support phone dialogues as real data, implement emotion recognition techniques on voice and text in order to predict the sentiment of customers. Since only small proportion of customers give feedbacks about their satisfaction rate, building this sentiment analysis system can help the organization understand the customer satisfaction rate spontaneously.

Facial expression of a person representative of similar emotions is not always unique. Naturally, the facial features of a subject taken from different instances of the same emotion have wider variations. In presence of two or more facial features, the variation of the attributes together makes the emotion recognition problem more complicated. This variation is the main source of uncertainty in the emotion recognition problem, which has been addressed here in two steps using type-2 fuzzy sets. First a type-2 fuzzy face-space is constructed with the background knowledge of facial features of different subjects for different emotions. Second, the emotion of an unknown facial expression is determined based on the consensus of the measured facial features with the fuzzy face-space. General Type-2 Fuzzy Sets have been used to model the fuzzy face space. The general type-2 fuzzy set involves both primary and secondary membership distributions, which have been obtained here by formulating and solving an optimization problem. The optimization problem here attempts to minimize the difference between two decoded signals: the first one being the type-1 defuzzification of the average primary membership distributions obtained from the n-subjects, while the second one refers to the type-2 defuzzified signal for a given primary distribution with secondary memberships as unknown. The uncertainty management policy adopted using general type-2 fuzzy set has resulted in a classification accuracy of 96.67%.

A challenging task in audiovisual emotion recognition is to implement neural network architectures that can leverage and fuse multimodal information while temporally aligning modalities, handling missing modalities, and capturing information from all modalities without losing information during training. These requirements are important to achieve model robustness and to increase accuracy on the emotion recognition task. A recent approach to perform multimodal fusion is to use the transformer architecture to properly fuse and align the modalities. This study proposes the AuxFormer framework, which addresses in a principled way the aforementioned challenges. AuxFormer combines the transformer framework with auxiliary networks. It uses shared losses to infuse information from single-modality networks that are separately embedded. The extra layer of audiovisual information added to our main network retains information that would otherwise be lost during training. The results show that the AuxFormer architecture achieves macro and micro F1Scores of 71.3% and 71.7%, respectively, on the CREMA-D corpus. For the MSP-IMPROV corpus, AuxFormer achieves a macro and micro F1-Scores of 70.4% and 76.5%, respectively. The results for both corpora are significantly better than strong baselines, indicating that our framework benefits from auxiliary networks. We also show that under non-ideal conditions (e.g., missing modalities) our architecture is able to sustain strong performance under audio-only and video-only scenarios, benefiting from a optimized training strategy.

The rapid growth of edge artificial intelligence (AI) and the escalating computational demands of multimodal perception systems accentuate the bottlenecks of von Neumann architecture and the high costs of training. Therefore, to streamline system construction, reduce energy consumption, and improve the processing efficiency of artificial perception systems, the advancement of emerging device technologies and next-generation computing architecture is urgently required. Here, we present a brain-inspired reservoir computing (RC) framework employing dynamic memristors with the Ag/WOx/ITO structure, enabling efficient audio-visual emotion recognition. The developed WOx-based memristor demonstrates key biological synaptic functionalities, including short-term plasticity, paired-pulse facilitation/depression, and nonlinear temporal dynamics, enabling its use as an effective physical reservoir. By exploiting the device's transient response to electrical pulses and its nonlinear mapping capability, complex spatiotemporal signals from auditory and visual modalities are transformed into separable high-dimensional reservoir states through simple 4-bit pulse sequences. The implemented dual-modal RC system achieves remarkable classification accuracies of 95.46% for speech-based emotion recognition and 91.08% for facial expression recognition, with an energy cost of only ∼3.9 nJ per pulse operation. This work provides a compelling hardware paradigm for low-power, real-time audio-visual perception on edge devices, paving the way for advanced neuromorphic computing.

This paper presents a neural network (NN) based multimodal fusion classifier for automatic emotion recognition. The audio and visual channels provide complementary information, so we utilize features from three behavioral cues: frontal-view facial expression, profile-view facial expression and vocalization (audio). The problem of interest is to recognize basic emotions, and we use dimensional representation of emotion as the heuristic information (hints) when training NNs used for both single cue processing and multimodal fusion. With the aid of hints, the weights of NNs could learn optimized feature groupings and recognition accuracy of our classifier would be notably improved even when training data is insufficient. Experimental results on audio-visual emotion data recorded by ourselves in Wizard of Oz scenarios and emotion data from the Semaine naturalistic database both demonstrate that our approach is effective and promising.

In human–computer interaction, emotion recognition provides a deeper understanding of the user’s emotions, enabling empathetic and effective responses based on the user’s emotional state. While deep learning models have improved emotion recognition solutions, it is still an active area of research. One important limitation is that most emotion recognition systems use only text as input, ignoring features such as voice intonation. Another limitation is the limited number of datasets available for multimodal emotion recognition. In addition, most published datasets contain emotions that are simulated by professionals and produce limited results in real-world scenarios. In other languages, such as Spanish, hardly any datasets are available. Therefore, our contributions to emotion recognition are as follows. First, we compile and annotate a new corpus for multimodal emotion recognition in Spanish (Spanish MEACorpus 2023), which contains 13.16 h of speech divided into 5129 segments labeled by considering Ekman’s six basic emotions. The dataset is extracted from YouTube videos in natural environments. Second, we explore several deep learning models for emotion recognition using text- and audio-based features. Third, we evaluate different multimodal techniques to build a multimodal recognition system that improves the results of unimodal models, achieving a Macro F1-score of 87.745%, using late fusion with concatenation strategy approach.

The effects of maternal exercise on fetal heart rate and movement patterns.