Speech represents a fundamental medium for conveying human emotions and, as a result, speech-based emotion recognition (SER) systems have become pivotal in advancing human-computer interaction (HCI) across a range of applications. While significant progress has been made in speech emotion recognition over recent years, existing solutions still face several key challenges, in that they: <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(i)$</tex-math></inline-formula> rely excessively on subjectively annotated (discrete) labels during training, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(ii)$</tex-math></inline-formula> often overlook the label ambiguity of speech samples that express more than one class of emotions, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(iii)$</tex-math></inline-formula> underutilize unlabeled or ambiguous speech, for which typically a label distribution (or so-called soft labels) is available. To address these issues, we propose in this paper a novel SER model that explicitly handles ambiguous speech samples and overcomes the shortcomings outlined above. Central to our approach is a novel real-time soft-label correction strategy designed to refine the annotations assigned to ambiguous speech. The proposed model leverages both, (explicitly) labeled as well as ambiguous samples and applies the dynamic soft-label correction strategy alongside an enhanced inter-class difference loss function to iteratively optimize the label distributions during training. We theoretically demonstrate that our method is capable of approximating the true emotional distribution of speech even in the presence of label noise, suggesting that utilizing ambiguous speech samples without explicit emotion labels still contributes toward more effective emotion recognition. Furthermore, we integrate the representational power of convolutional neural networks (CNNs) with the contextual modeling capabilities of Wav2Vec 2.0 to enable a comprehensive extraction of spatio-temporal speech features. Experimental results on the IEMOCAP multi-label dataset confirm the effectiveness of our approach, achieving state-of-the-art performance with significant improvements in weighted accuracy (WA) and unweighted accuracy (UA) over competing methods.

Emotion and intent joint understanding in multimodal conversations (MC-EIU) aims to model the semantic dependencies among multimodal conversations while inferring emotion and intent information. Despite making progress, existing methods overlook the differential contributions of modalities and rely on single-round interactions for emotion and intent recognition, resulting in suboptimal model understanding performance. To overcome these limitations, we propose MEIPro, a novel progressive interaction and adaptive weight fusion based multimodal joint understanding of emotion and intent framework. We first design a hierarchical denoising module to effectively remove noise and redundant information from multimodal data. Then, we propose an adaptive weight fusion mechanism that dynamically fuses multimodal features by taking the true classification probabilities of each modality as their respective contributions, thus enhancing the fusion process. Additionally, we present a progressive dual task interaction module to capture the deep seated interactions between emotion and intent through a step-by-step multi-round iteration. Experiments on the benchmark MC-EIU bilingual dataset demonstrate that our MEI-Pro framework significantly outperforms state-of-the-art baselines in both emotion and intent tasks. Specifically, on the English dataset, the F1-scores of the multimodal emotion and intent understanding tasks have increased by 6.12% and 7.25% respectively.

Human emotion is expressed in many communication modalities and media formats and so their computational study is equally diversified into natural language processing, speech processing, computer vision, etc. Similarly, a large variety of representation formats has been used in previous research to describe emotions (polarity scales, basic emotion categories, dimensional approaches, appraisal theory, etc.) and has led to an ever proliferating diversity of datasets, predictive models, and software tools for emotion analysis. Because of these two distinct types of heterogeneity, at the expressional and representational level, there is a dire need to unify previous work on increasingly diverging data and label types. This article presents such a unifying computational model. We propose a training procedure that learns a shared latent representation for emotions, so-called emotion embeddings, independent of different natural languages, communication modalities, media or representation label formats, and even disparate model architectures. Experiments on a wide range of heterogeneous affective datasets indicate that this approach yields the desired interoperability for the sake of reusability, interpretability and flexibility, without penalizing prediction quality. Code and data are archived under DOI:10.5281/zenodo.7405327.

Dynamic facial expression recognition (DFER) infers emotions from the temporal evolution of expressions, unlike static facial expression recognition (SFER), which relies solely on a single snapshot. This temporal analysis provides richer information and promises greater recognition capability. However, current DFER methods often exhibit unsatisfied performance largely due to fewer training samples compared to SFER. Given the inherent correlation between static and dynamic expressions, we hypothesize that leveraging the abundant SFER data can enhance DFER. To this end, we propose Static-for-Dynamic (S4D), a unified dual-modal learning framework that integrates SFER data as a complementary resource for DFER. Specifically, S4D employs dual-modal self-supervised pre-training on facial images and videos using a shared Vision Transformer (ViT) encoder-decoder architecture, yielding improved spatiotemporal representations. The pre-trained encoder is then fine-tuned on static and dynamic expression datasets in a multi-task learning setup to facilitate emotional information interaction. Unfortunately, vanilla multi-task learning in our study results in negative transfer. To address this, we propose an innovative Mixture of Adapter Experts (MoAE) module that facilitates task-specific knowledge acquisition while effectively extracting shared knowledge from both static and dynamic expression data. Extensive experiments demonstrate that S4D achieves a deeper understanding of DFER, setting new state-of-the-art performance on FERV39K, MAFW, and DFEW benchmarks, with weighted average recall (WAR) of 53.65%, 58.44%, and 76.68%, respectively. Additionally, a systematic correlation analysis between SFER and DFER tasks is presented, which further elucidates the potential benefits of leveraging SFER.

The rapid development of natural language processing (NLP) has enabled automatic emotion recognition in conversations (ERC). However, most existing models train on isolated one-hot emotion labels that overlook emotional connections between utterances and cannot effectively convey the extent and complexity of nuanced emotional perception similar to human understanding, resulting in reduced performance. To address this, we propose a self-supervised <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</b>ituation-aware <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">E</b>motion <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</b>abel (SEL) generation method, which can automatically and effectively compute weights of all emotions for every utterance in a self-supervised way, utilizing no extra emotional annotated data. To generate SELs, we propose a self-supervised situation-aware method to first fuse <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">situation</i> information into utterances to generate emotion-perceptive representations. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Situation</i> is defined as the combination of the topic and surrounding people, with which emotions are widely thought to be more predictable. Then, we combine similarities of the emotion-perceptive representations with prior human knowledge to compute SELs. SEL can be deployed in most existing ERC models that train on one-hot emotion labels to enhance their performance. Experiments on six ERC models and four benchmark datasets indicate that SEL can effectively boost the performance of base models and attain state-of-the-art outcomes. Ablation study, case study, and quantitative analysis further show how SEL works.

Emotion recognition from wearable physiology could enable timely, personalized support when behavioural cues are atypical or self-report is limited. This need is particularly acute for students with Autism Spectrum Disorder (ASD) during Applied Behavior Analysis (ABA) therapy, where therapists must often infer internal states indirectly and in real time. Although Transformer architectures excel at modelling complex temporal dependencies, most physiological emotion-recognition models are developed on neurotypical cohorts and are rarely evaluated under therapy-relevant validation protocols or integrated into operational ABA workflows. Here, we show that a supervised Transformer operating on continuous 1D peripheral physiological time series supports practical emotion recognition for ASD intervention and is associated with measurable improvements in therapy. Using therapist annotations strictly as training targets (not as input features), we benchmarked the model on an expert-curated dataset of 1,459,337 therapist-labelled samples from <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n=12$</tex-math></inline-formula> students with ASD collected during authentic sessions, achieving Leave-One-Session-Out (LOSecO) subject-specific accuracies of 84.28% (valence), 85.73% (arousal), and 80.26% (5-class emotion). In an independent user-study cohort (<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N=33$</tex-math></inline-formula>, 1,722 sessions), integrating the system into ABA sessions was associated with statistically significant improvements in students' performance compared with standard practice. The system is further deployed in a production ABA platform serving over 2,000 students across 88 sites in Hong Kong, Singapore, and Canada, with 18 ms per-window inference latency and 99.97% uptime over three months. Our results demonstrate that Transformer-based physiological emotion recognition can be implemented reliably within the constraints of large-scale special education platforms, supporting practical translation from algorithmic performance to real-world intervention settings.

The development of driver monitoring systems (DMS) has emerged as a pivotal advancement in automotive safety to monitor driver behaviors and prevent road accidents. Driver emotions are one of the critical factors that affect driving safety. While numerous studies have explored facial macro-expressions (FMaEs) of drivers, these facial expressions can be consciously controlled, allowing individuals to conceal their true negative emotions. However, inaccurate capturing of true emotions can lead to severe accidents. In this paper, we focus on facial micro-expressions (FMEs), which are brief and involuntary facial movements that reveal deeper insights into genuine emotions. Thereby, they provide a more accurate measure of a driver's emotional condition, and are crucial for assessing driver readiness and safety. To facilitate FME studies of drivers, we analyzed driver videos from an affective driver-pedestrian interaction experiment, providing corresponding facial action units and FME annotations, and built a real driver FME dataset. By analyzing the driver data, we found that there are more FMEs when drivers respond to negative stimuli compared with positive stimuli, and there are more FMaEs when drivers respond to positive stimuli compared with negative stimuli. To protect driver identities while maintaining the integrity of the facial movements, we released a synthesized driver FME dataset. Furthermore, we benchmarked deep learning-based architectures on the real and synthetic driver FME datasets, uncovering the challenges inherent in recognizing FMEs in real-world driving scenarios.

Multimodal annotations provide important cues for understanding how a conversation proceeded, particularly in relation to affective factors. In this paper, we extend the automated conversation annotation system MONAH with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">pitch</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">volume</i> annotations to encode new affective cues, making MONAHv3 the state-of-the-art automatic annotation system in terms of the number of automatically annotated aspects. MONAHv3 provides an automated solution that, while not a direct equivalent, offers performance that is competitive with the manually produced Jefferson transcription system. In automatic evaluations, the additions significantly improve supervised learning in ten out of eighteen experiments. In human evaluations of emotion recognition, the additions significantly outperformed the Jefferson transcription system. Usability studies further showed that MONAHv3 is much more user-friendly than Jefferson-style transcripts. Lastly, in human evaluations of paralinguistic cues (e.g., tone and volume), MONAHv3 achieved performance comparable to the Jefferson system, while remaining more competitive in kinesics (e.g., describing actions). This distinction between automated and manual systems is critical for appreciating the scalability and accessibility that MONAHv3 offers.

How people evaluate, select, and engage with others in cooperative settings significantly impacts their well-being, happiness, and success. However, navigating these processes is complex. Equipping systems with the ability to recognize, interpret, and even engage during such socio-cognitive processes can increase their potential to support humans in these socio-cognitive processes and be more successful in adjusting to the social environment they are embedded in (e.g., understanding human preferences and attitudes), leading to better quality interactions and decision-making for future partners. Yet, the developments of such systems depend on available datasets. However, based on our knowledge, no dataset exists that can be used to model partner selection for joint tasks. To support research focused on creating such intelligent systems, we introduce the PARSEL dataset – a comprehensive corpus of dyadic interactions designed for computational modeling of PARtner SELection processes and collaborative behavior. In total, 297 participants took part in the datasets. The dataset contains measurements of partner selection decisions over three different stages, as well as factors that may influence partner selection in the context of (online) social interactions. It includes audiovisual recordings that offer fine-grained behavioral cues used during these interactions, self-reported traits, and reported perceptions of person-, situation- and team-specific phenomena. By providing this resource, we aim to foster advancements in computational methods that can effectively model and augment socio-cognitive processes, contributing to socially aware intelligent systems and enhanced human-system interactions.