Attention-Guided Network Model for Image-Based Emotion Recognition

Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network-time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31-50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

Introducing a novel dataset for facial emotion recognition and demonstrating significant enhancements in deep learning performance through pre-processing techniques

FBSA-Net: A novel model based on attention mechanisms for emotion recognition in VR and 2D scenes

ObjectiveIn the field of digital health, emotion recognition is vital for human–computer interaction and clinical neuropsychology. However, existing deep-learning models often lack interpretability, limiting their ability to provide insights into brain activity.MethodsThis study proposes an innovative brain-inspired neuronal competition model for emotion recognition from electroencephalography (EEG). First, a discrete wavelet transform is employed to extract five brain rhythms (delta, theta, alpha, beta, and gamma) as feature inputs, maximizing the preservation of signal information. Then, the proposed model simulates excitatory and inhibitory competitive dynamics between neurons, where a rhythm attention encoder dynamically focuses on key brain rhythms and uses a leaky integrate-and-fire model to simulate the neuronal decision-making process. Finally, emotion recognition is implemented based on a winner-take-all principle.ResultsTen-fold cross-validation experiments on three widely used datasets, DEAP, SEED, and DREAMER, achieve average accuracies of 86.64–92.70% on DEAP, 81.89% on SEED, and 80.38–82.70% on DREAMER, respectively. These results indicate that different brain rhythms exhibit distinct characteristics in various emotion recognition tasks.ConclusionsThis study presents a biologically plausible paradigm for EEG emotion recognition, laying the foundation for understanding the neurophysiological basis of brain activity and developing reliable tools for healthcare applications, such as mental health monitoring and depression detection.

Accurate segmentation is a basic and crucial step for medical image processing and analysis. In the last few years, U-Net, and its variants, have become widely adopted models in medical image segmentation tasks. However, the multiple training parameters of these models determines high computation complexity, which is impractical for further applications. In this paper, by introducing depthwise separable convolution and attention mechanism into U-shaped architecture, we propose a novel lightweight neural network (DSCA-Net) for medical image segmentation. Three attention modules are created to improve its segmentation performance. Firstly, Pooling Attention (PA) module is utilized to reduce the loss of consecutive down-sampling operations. Secondly, for capturing critical context information, based on attention mechanism and convolution operation, we propose Context Attention (CA) module instead of concatenation operations. Finally, Multiscale Edge Attention (MEA) module is used to emphasize multi-level representative scale edge features for final prediction. The number of parameters in our network is 2.2 M, which is 71.6% less than U-Net. Experiment results across four public datasets show the potential and the dice coefficients are improved by 5.49% for ISIC 2018, 4.28% for thyroid, 1.61% for lung and 9.31% for nuclei compared with U-Net.

In recent years, attention mechanisms have shown great potential in various computer vision tasks. However, most existing methods focus on developing more complex attention modules for better performance, which inevitably increases the complexity of the model. To overcome performance and complexity tradeoffs, this paper proposes efficient spatial and channel attention (ESCA), a symmetric, comprehensive, and efficient attention module. By analyzing squeeze-and-excitation (SE), convolutional block attention module (CBAM), coordinate attention (CA), and efficient channel attention (ECA) modules, we abandon the dimension-reduction operation of SE module, verify the negative impact of global max pooling (GMP) on the model, and apply a local cross-channel interaction strategy without dimension reduction to learn attention. We not only care about the channel features of the image, we also care about the spatial location of the target on the image, and we take into account the effectiveness of channel attention, so we designed the symmetric ESCA module. The ESCA module is effective, as demonstrated by its application in the ResNet-50 classification benchmark. With 26.26 M parameters and 8.545 G FLOPs, it introduces a mere 0.14% increment in FLOPs while achieving over 6.33% improvement in Top-1 accuracy and exceeding 3.25% gain in Top-5 accuracy. We perform image classification and object detection tasks on ResNet, MobileNet, YOLO, and other architectures on popular datasets such as Mini ImageNet, CIFAR-10, and VOC 2007. Experiments show that ESCA can achieve great improvement in model accuracy at a very small cost, and it performs well among similar models.

Human action recognition is a research hotspot in the field of ambient intelligence, serving as a foundation for the Internet of Healthcare Things (IoHT) and smart home wellness, with extensive application value. Currently, most research primarily focuses on the performance of single-modal approach for human action recognition. However, single-modal data cannot adequately capture the action characteristics of the human body. For instance, camera data lacks detailed micro-motion information, while radar data lacks visual appearance information. This limitation makes human action recognition systems susceptible to complex covariate influences. To address this issue, this paper proposes a Multi-Modal Fusion Model, M2FM, to accurately recognize various complex human action from millimeter-wave radar signals and video data. In the millimeter-wave radar branch, a LFNet network is constructed to capture richly hierarchical human action representation by extracting micro-Doppler feature and cadence velocity feature simultaneously. Specially, a Linear Feedforward Neural Network (LFNN) module is designed for modeling global-local feature of human action. In the video branch, a lightweight Transformer-based video analysis and action recognition network, STL-Former, is developed. Specially, an Agent downSampling (AS) attention module and a Linear Maxpooling (LM) attention module are designed to achieve context-aware downsampling with global receptive field and spatial vector sequence downsampling, respectively. Additionally, a multi-scale Linear attention (Litner) module is designed for modeling the spatio-temporal information of human action. Finally, the outputs from two branches are fused by a filter-based Dempster-Shafer theory. The experimental results on a self-collected Multi-modal Human Action Dataset, JH-MHAD, show that the M2FM model outperforms other advanced models, with an average recognition accuracy of 99.3% for eight different human actions, and it demonstrates strong adaptability under different conditions.

Emotion is the psychological and physiological response of a person or animal to a specific stimulus or event, and it can also have an impact on cognition, decision-making, and behavior. Research on emotion recognition has wide applications in various fields, such as dialogue and communication with machines, and in the medical field. At present, most emotion recognition systems are based on PC or embedded platforms. Compared with traditional emotion recognition platforms, FPGA has good parallel computing ability and can process multiple data streams simultaneously, making it suitable for complex pattern recognition in emotion recognition. This article chooses ZYNQ as the development platform, combined with the collaborative ability of FPGA and ARM processors, to build a complete hardware application system that can carry neural networks. Secondly, in response to the problems of complex computation and high storage resource consumption in existing neural network computing frameworks, this paper adopts a lightweight neural network MobileNetV3 and implements and improves it. The model is encapsulated and ported using the Vivado HLS development tool and C++. The hardware IP core is called on the hardware platform to achieve ZYNQ based facial emotion classification and recognition, and the power consumption of hardware resources is analyzed.

A novel facial emotion recognition method for stress inference of facial nerve paralysis patients

Atypical and subtle facial expression patterns in individuals with autism spectrum disorder (ASD) pose a significant challenge for automated emotion recognition. This study evaluates and compares the performance of convolutional neural networks (CNNs) and transformer-based deep learning models for facial emotion recognition in this population. Using a labeled dataset of emotional facial images, we assessed eight models across four emotion categories: natural, anger, fear, and joy. Our results demonstrate that transformer models consistently outperformed CNNs in both overall and emotion-specific metrics. Notably, the Swin Transformer achieved the highest performance, with an accuracy of 0.8000 and an F1-score of 0.7889, significantly surpassing all CNN counterparts. While CNNs failed to detect the fear class, transformer models showed a measurable capability in identifying complex emotions such as anger and fear, suggesting an enhanced ability to capture subtle facial cues. Analysis of the confusion matrix further confirmed the transformers’ superior classification balance and generalization. Despite these promising results, the study has limitations, including class imbalance and its reliance solely on facial imagery. Future work should explore multimodal emotion recognition, model interpretability, and personalization for real-world applications. Research also demonstrates the potential of transformer architectures in advancing inclusive, emotion-aware AI systems tailored for autistic individuals.

Emotion identification plays a vital role in human interactions. For this purpose, Computer-vision methods for automatic emotion recognition is nowadays a widely studied topic. One of the most studied approaches for automatic emotion recognition is processing multi-channel Electroencephalogram signals (EEG). This paper presents a new model for emotion recognition using brain maps as input and providing emotion states in terms of arousal and valence as output. Brain maps are a spatial representation of features extracted from EEG signals. The proposed model, called Multi-Task Convolutional Neural Network (MT-CNN), is fed with stacked brain maps of four different waves of different frequency bands: alpha, beta, gamma and theta, using differential entropy and power spectra density and considering observation windows of 0.5s. This model is trained and tested on the DEAP dataset, a well-known dataset for comparison purposes. This work shows that the MT-CNN nerforms better than other methods.

Continuous emotion recognition is a challenging task and a key part of human-computer interaction, especially multimodal emotion recognition can effectively improve the accuracy and robustness of recognition. But there are limited emotion data sets, and it is difficult to extract emotion features. We present a multi-level segmented decision level fusion emotion recognition model to improve the performance of emotion recognition. In this paper, we predict multi-modal dimensional emotional state on AVEC2017 dataset. Our model uses Bidirectional Long Short-Term Memory (BLSTM) as multi-level segmented emotional feature learning model, and uses the SVR model as fusion model of the decision layer. The BLSTM can model different forms of emotional information in time, and can also consider the impact of previous and later emotional features on current results. The SVR model can compensate for the redundant information of emotion recognition. At the same time, we also consider annotation delay and temporal pooling in our multi-modal dimensional emotion recognition model. Our multi-modal emotion recognition model achieves significant recognition improvements and provide the robustness. Finally, we compare the baseline methods which used the same dataset, and find that the CCC performance of our method is the best on arousal, which is 0.685. Our research shows that the proposed multi-layer segmentation decision level fusion emotion recognition model is conducive to improving performance.

Facial Emotion Recognition (FER) is crucial for applications such as human-computer interaction and mental health diagnostics. This study presents the first empirical comparison of open-source Vision-Language Models (VLMs), including Phi-3.5 Vision and CLIP, against traditional deep learning models—VGG19, ResNet-50, and EfficientNet-B0—on the challenging FER-2013 dataset, which contains 35,887 low-resolution, grayscale images across seven emotion classes. To address the mismatch between VLM training assumptions and the noisy nature of FER data, we introduce a novel pipeline that integrates GFPGAN-based image restoration with FER evaluation. Results show that traditional models, particularly EfficientNet-B0 (86.44%) and ResNet-50 (85.72%), significantly outperform VLMs like CLIP (64.07%) and Phi-3.5 Vision (51.66%), highlighting the limitations of VLMs in low-quality visual tasks. In addition to performance evaluation using precision, recall, F1-score, and accuracy, we provide a detailed computational cost analysis covering preprocessing, training, inference, and evaluation phases, offering practical insights for deployment. This work underscores the need for adapting VLMs to noisy environments and provides a reproducible benchmark for future research in emotion recognition.

In recent years, Electroencephalogram (EEG)-based emotion recognition has developed rapidly and gained increasing attention in the field of brain-computer interface. Relevant studies in the neuroscience domain have shown that various emotional states may activate differently in brain regions and time points. Though the EEG signals have the characteristics of high temporal resolution and strong global correlation, the low signal-to-noise ratio and much redundant information bring challenges to the fast emotion recognition. To cope with the above problem, we propose a Temporal and channel Transformer (TcT) model for emotion recognition, which is directly applied to the raw preprocessed EEG data. In the model, we propose a TcT self-attention mechanism that simultaneously captures temporal and channel dependencies. The sliding window weight sharing strategy is designed to gradually refine the features from coarse time granularity, and reduce the complexity of the attention calculation. The original signal is passed between layers through the residual structure to integrate the features of different layers. We conduct experiments on the DEAP database to verify the effectiveness of the proposed model. The results show that the model achieves better classification performance in less time and with fewer resources than state-of-the-art methods.

Electroencephalogram (EEG)-based analysis of emotional states is important for health assessment, emotion monitoring, and treatment support. Accurate EEG emotion recognition depends on both rich emotional features and well-designed recognition models. While deep learning can automatically extract features and classify data effectively, it often fails to produce diverse emotional EEG features, and manually designing network structures is time-consuming. Neural architecture search (NAS) helps find the best network design and reduces manual effort. In this work, we propose an attention-based convolutional neural network (CNN) architecture search method using spatial-spectral features for EEG emotion recognition, called ACAS-EER. It is optimized to better match emotional EEG characteristics. Firstly, differential entropy (DE) and power spectral density (PSD) were combined to create EEG feature maps, providing more spatial and spectral information. Secondly, a search space with four lightweight attention modules was designed. The final architecture, found automatically, includes convolution, pooling, and multiple attention modules. Also, an operation-level Dropout method was used to avoid poor performance caused by too many parameterless operations. On the DEAP dataset, ACAS-EER achieved high average accuracy and F1 scores (97.40% and 97.33% for valence; 97.68% and 97.35% for arousal). It performed better than most hand-designed deep learning models and other NAS models.