The rapid development of mobile internet has turned multimodal sentiment analysis (MSA) into a prominent research focus. Despite the progress achieved by existing models, the heterogeneity problem resulting from modality differences presents significant challenges to MSA. Furthermore, while much focus has been placed on dynamic learning within a single sample, the learning of inter-sample and inter-class relationships has been largely overlooked. To tackle these problems, we propose a multi-scale text-guided progressive fusion network. Initially, a domain adaptation strategy is employed to fine-tune BERT model, enhancing its support for downstream tasks. Subsequently, we design a progressive fusion module that centers on text as the core modality, guiding the integration of audio and visual modalities to generate more enriched derivative feature representations. Within the cross-modal transformer, the integration of core and super-modal features captures fine-grained cross-modal dependencies. Furthermore, we introduce dual contrastive learning to facilitate the modeling of inter-sample and cross-category relationships, which further alleviates the impact of distributional discrepancies among modalities. Experimental results on the widely used CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets demonstrate that the proposed method outperforms existing methods in multimodal sentiment classification tasks. The method achieved sentiment binary classification accuracies of 84.90%, 85.03%, and 81.76% on these datasets, and its effectiveness is further validated through multiple experiments.

Understanding emotional states is fundamental to advancing next-generation AI systems with human-like attributes. Combining the complementary strengths of electroencephalog raphy (EEG) and facial expressions holds great promise for advancing multimodal emotion recognition (MER). EEG provides objective measurements of neural activity, while facial expressions convey rich, externally observable emotional cues. However, existing joint learning frameworks often fall short of fully exploiting the synergy between these modalities. Two key challenges remain unresolved: (1) insufficient cross-modal alignment and interaction prior to fusion which limits the semantic complementarity between modalities; and (2) modality unreliability caused by temporal fluctuations in signal quality and inconsistencies in emotional semantics. To address these limitations, we propose a novel framework that integrates a Step-wise Prompts (SwiP) module with an Uncertainty-Aware Dynamic Fusion (UADF) mechanism. SwiP enables progressive, fine-grained interaction by introducing sequential facial features as visual prompts to guide EEG representation learning, thereby enhancing cross modal complementarity. UADF dynamically adjusts modality contributions through a token- and modality-level uncertainty estimation scheme, enabling the model to selectively emphasize informative inputs and suppress noisy or irrelevant signals. Extensive experiments on benchmark datasets demonstrate that our method achieves state-of-the-art performance, consistently outperforming competitive baselines in both accuracy and stability. These results highlight the potential of our framework as a robust and generalizable solution for real-world affective computing applications.

This paper introduces a novel dynamic graph learning approach for frequency graphs, underpinned by a suite of baseline methodologies and the Multi-scale Controllable Graph Convolutional Network (MCGC-net), a multi-channel dynamic graph learning framework. This approach addresses the limitations of temporal graphs in capturing global and local correlations in music signals for music emotion recognition tasks. Frequency graphs integrate timbre and chroma audio features to capture both local and global correlations in music, enhancing emotion classification and segmentation. The graph feature extractor module improves the dissemination of hierarchical feature information through control units and enhances the depiction of remote node attributes via shortcut connections. Additionally, the multi-channel feature extractor module expands the receptive field, enhancing inter-channel feature extraction. The integration of these modules enables the model to proficiently extract musical frequency features, uncovering latent correlations among various frequencies and their impact on emotion recognition. Empirical validation across the 1000songs, PMEmo, and Memo2496 datasets demonstrates the strategy's efficacy, with MCGC-net achieving classification accuracies of 79.03% for Arousal and 70.05% for Valence on 1000songs; 85.81% and 75.42% on PMEmo; and 79.52% and 78.61% on Memo2496, respectively. Complementary ablation studies and T-order robustness assessments further validate the unique contributions of model components and the performance enhancements provided by shortcut connections.

Touch Emotion Recognition (TER) is an emerging field in affective computing that focuses on interpreting human emotions through touch interactions. This survey provides a comprehensive review of the haptic technologies, methodologies, and applications of TER, emphasizing its role as a rich but underexplored modality for emotion recognition. A new taxonomy for TER is proposed, categorizing research based on recognition entities, recognition approaches, and touch gesture types, offering a structured perspective for advancing TER studies. The survey highlights the importance of affective touch gesture datasets, evaluates the hardware technologies underpinning TER systems, and reviews key methods for data processing and classification. Applications across domains such as emotional AI, social robotics, virtual reality, and healthcare are examined, showcasing the potential of TER in enhancing the emotional intelligence of artificial agents and robotic systems. Finally, challenges including dataset scarcity, device standardization, and cultural variability are discussed, alongside future directions for creating more robust and inclusive TER systems.

This work presents a systematic literature review of deep learning techniques for text-based emotional response generation, a particular topic in the area of emotional dialogue systems. In particular, this work focus on systems that support text-based interaction where the system detects the user's emotion and responds accordingly. We have scouted over 6000 articles for this review, selecting 74 of them and reviewing them to identify datasets, methods, evaluation techniques, and challenges in the area. Our analysis points out an overwhelming bias toward researching these problems in English and Chinese, with few to no resources for other languages. Although several methodologies are proposed in the literature, including Seq2Seq and Conditional Variational Autoencoders, more recent research has been focused on the Transformer architecture. It is recognized that there are manual metrics that do not have an equivalent to automatic metrics, and there are variations in human evaluation results in comparison to modern automated metrics. With this paper, we aim to summarize the state-of-the-art of deep learning techniques for emotional response generation while identifying open research questions.

Increasing access to affordable physiological sensors and computing devices has bolstered the development of emotionaware systems that recognize human emotions. However, to date, there has been little research testing machine-driven approaches to recognizing boredom, particularly in learning contexts. To address that, this study tests the automatic recognition of boredom induced during a video lecture using a multimodal recognition system relying on physiological signals. Electroencephalogram (EEG), electrocardiogram (ECG), galvanic skin response (GSR), and eye gaze data were recorded from 84 healthy adults (mean age = 26.90 ± 5.29) as they watched both non-boring and boring educational videos to control and induce boredom. Signal features were extracted from the physiological data and validated using Wilcoxon signed-rank tests prior to evaluating their utility for boredom recognition with three separate machine-learning classification techniques, namely Extreme Gradient Boosting (XGB), Random Forest (RF), and Gradient Boosting (GB), with leave-one-out cross-validation. Both unimodal and multimodal recognition systems were assessed by evaluating model performance using each physiological signal separately or in combination (using decision fusion). Multimodal recognition was found to enhance model performance when compared to any single modality, with the highest average boredom recognition accuracy of 88.56% ± 0.82% recorded for EEG + eye gaze modal fusion with RF.

People often stick to their existing beliefs, ignoring contradicting evidence or only interacting with those who reinforce their views. This tendency, referred to as being rigid in one's beliefs, is often driven by one's emotional connection to their beliefs and worsened by social media platforms which promote highly personalized content to maximize user engagement. As increased belief rigidity can negatively impact decisions, it is crucial to study and design interventions to reduce belief rigidity in online social networks. We design an experimental framework to passively infer belief rigidity and empirically quantify the effects of introducing randomness into the network structure as an intervention strategy. We recruit 163 participants into two conditions: one emulating traditional online platforms and the other more random. Our results suggest that individuals' beliefs are influenced by peer opinions, regardless of whether those opinions are similar to or differ from their own. Moreover, people are more likely to incorporate peers with (slightly) differing opinions into their networks when the recommendations are more randomized. Our findings highlight the need for future exploration into how design interventions can shape people's psycho-social functionalities in online social platforms.

Micro-gestures are unconsciously performed body gestures that can convey the emotion states of humans and start to attract more research attention in the fields of human behavior understanding and affective computing as an emerging topic. However, the modeling of human emotion based on micro-gestures has not been explored sufficiently. In this work, we propose to recognize the emotion states based on the micro-gestures by reconstructing behavioral patterns with a hypergraph-enhanced Transformer in a hybrid-supervised framework. In the framework, hypergraph Transformer based encoder and decoder are separately designed by stacking the hypergraph-enhanced self-attention and multiscale temporal convolution modules. Especially, to better capture the subtle motion of micro-gestures, we construct a decoder with additional upsampling operations for a reconstruction task in a self-supervised learning manner. We further propose a hypergraph-enhanced self-attention module where the hyperedges between skeleton joints are gradually updated to present the relationships of body joints for modeling the subtle local motion. Lastly, for exploiting the relationship between the emotion states and local motion of micro-gestures, an emotion recognition head from the output of encoder is designed with a shallow architecture and learned in a supervised way. The end-to-end framework is jointly trained in a one-stage way by comprehensively utilizing self-reconstruction and supervision information. The proposed method is evaluated on two publicly available datasets, namely iMiGUE and SMG, and achieves the best performance under multiple metrics, which is superior to the existing methods. The code is available on Github (https://github.com/xiazhaoqiang/H2OFormerMicroGestureRec).

Sensory substitution devices (SSDs) enable the compensation of sensory loss by translating information from one modality to another. However, current SSDs face limitations in conveying emotional content, which reduces user engagement and acceptance. This study investigates emotional coherence in visual-to-auditory sensory substitution, focusing on the mapping between visual stimuli and musical features. We introduce a novel experimental protocol that systematically measures emotional responses to various combinations of visual and auditory stimuli monitoring facial expression and physiological parameters, and using the Self-Assessment Manikin (SAM) scale. The collected database, containing data from 36 participants, is analysed to answer three research questions: 1) explore the role of valence and arousal dimensions in driving associations between visual content and musical characteristics, 2) verify emotional coherence between the SAM and the collected data, and 3) predict appropriate musical features for visual stimuli based on emotional response data. Results from our analyses revealed a strong preference for valence-matched stimuli over arousal-matched alternatives, with participants selecting valence-matched options in 83.0% of cases, compared to only 6.0% for arousal-matched stimuli. Furthermore, classical machine learning algorithms were successfully employed to classify high and low scores of the SAM metrics using the multimodal behavioural data, achieving 75.4% accuracy for dominance and 63.4% for arousal. Finally, the developed predictive models show promising results in predicting the selected song, with 67.9% in accuracy, and yield exceptionally low error rates in predicting musical characteristics, with mean squared errors reaching approximately <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$10^{-29}$</tex-math></inline-formula> for some musical features. These results provide a foundation for developing emotion-aware sensory substitution systems that maintain emotional congruence in the translation from visual to auditory modalities, potentially enhancing user engagement and acceptance of SSDs.

Comprehensive cognitive-affective profiling for precision ASD assessment is hindered by inadequate computational modeling of emotion-behavior dynamics. This gap persists due to limited methods for reconciling EEG, eye-tracking, and facial expression modalities-critical for capturing ASD heterogeneity drivers. This study proposes the Cognitive-Affective Ability Assessment (CA<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup>) framework, focusing on attention control in conjunction with valence-arousal, to address these limitations through: (1) <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Ability-Specific Neural Disentanglement</b> extracting attention- and emotion-specific representations from EEG using ET and facial expression recordings as anchors; (2) <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Symmetrical Cross-Ability Alignment</b> modeling contextual dependencies via a symmetric cross-attention mechanism; and (3) <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Uncertainty-Aware Ability Integration</b> fusing per-ability predictions using Dirichlet modeling and Dempster-Shafer theory while quantifying confidence. Extensive experiments have been conducted to evaluate CA<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{3}$</tex-math></inline-formula> vs. state-of-the-arts counterparts on the multimodel datasets (EEG, ET and facial expression recordings) with CCNU (27 ASD vs. 30 typically developing (TD) children) and BNU (49 ASD vs. 48 TD), and the results demonstrate that: 1) In the ASD cognitive-affective abilities assessment task, accuracy is improved by 2.7% for attention control and 2.5% for emotion perception, and by 3.5% for cognitive-affective abilities assessment, compared with current state-of-the-art methods. 2) For attention control and valence-arousal, composite ability gaps are measured between ASD and typically developing children: 15.6% and 19.9% for mild ASD, 28.2% and 39.1% for moderate ASD, and 44.4% and 61.3% for severe ASD, respectively. Overall, the framework effectively bridges computational assessment with clinically actionable insights, enabling robust decision-making through personalized cognitive-affective ASD profiles.