Artificial Intelligence and Technology

Advancements in neuroscience and deep learning have significantly enhanced bio-signal-based emotion recognition, a critical component in Brain-Machine Interface (BMI) applications for healthcare, human-computer interaction, and human-AI assistant communication. Former studies have proposed Manual Mapping electrode matrices and employing Convolutional Neural Networks (CNNs) to recognize spatial EEG activities. However, this Manual Mapping of EEG electrodes onto matrix grids limits spatial precision and introduces inefficiencies. This study proposes automated channel mapping methods of Orthographic Projection and Stereographic Projection to address these challenges, using Differential Entropy and Power Spectral Density with Linear Dynamical Systems as features. A 3-branch multiscale CNN was trained on open-source dataset, employing a 5-fold cross-classification approach. Experimental results demonstrate that higher-resolution grids (16×16, 24×24) with automated projections significantly outperform Manual Mappings, achieving up to a 4.06% improvement in classification accuracy (p < 0.05). This result indicates that enhancing spatial precision of EEG data improves emotion classification, establishing automated spatial mapping as an advancement in EEG-based emotion recognition.Clinical Relevance-Advancement in emotion classification accuracy can facilitate more reliable diagnostic tools and personalized therapeutic interventions for mental health disorders, such as depression and anxiety.

Emotions expressed by humans can be identified from facial expressions, speech signals, or physiological signals. Among them, the use of physiological signals for emotion classification is a notable emerging area of research. In emotion recognition, a person’s electrocardiogram (ECG) and galvanic skin response (GSR) signals cannot be manipulated, unlike facial and voice signals. Moreover, wearables such as smartwatches and wristbands enable the detection of emotions in people’s naturalistic environment. During the COVID-19 pandemic, it was necessary to detect people’s emotions in order to ensure that appropriate actions were taken according to the prevailing situation and achieve societal balance. Experimentally, the duration of the emotion stimulus period and the social and non-social contexts of participants influence the emotion classification process. Hence, classification of emotions when participants are exposed to the elicitation process for a longer duration and taking into consideration the social context needs to be explored. This work explores the classification of emotions using five pretrained convolutional neural network (CNN) models: MobileNet, NASNetMobile, DenseNet 201, InceptionResnetV2, and EfficientNetB7. The continuous wavelet transform (CWT) coefficients were detected from ECG and GSR recordings from the AMIGOS database with suitable filtering. Scalograms of the sum of frequency coefficients versus time were obtained and converted into images. Emotions were classified using the pre-trained CNN models. The valence and arousal emotion classification accuracy obtained using ECG and GSR data were, respectively, 91.27% and 91.45% using the InceptionResnetV2 CNN classifier and 99.19% and 98.39% using the MobileNet CNN classifier. Other studies have not explored the use of scalograms to represent ECG and GSR CWT features for emotion classification using deep learning models. Additionally, this study provides a novel classification of emotions built on individual and group settings using ECG data. When the participants watched long-duration emotion elicitation videos individually and in groups, the accuracy was around 99.8%. MobileNet had the highest accuracy and shortest execution time. These subject-independent classification methods enable emotion classification independent of varying human behavior.

Automated accurate emotion recognition system using rhythm-specific deep convolutional neural network technique with multi-channel EEG signals

The electroencephalogram (EEG) has great attraction in emotion recognition studies due to its resistance to deceptive actions of humans. This is one of the most significant advantages of brain signals in comparison to visual or speech signals in the emotion recognition context. A major challenge in EEG-based emotion recognition is that EEG recordings exhibit varying distributions for different people as well as for the same person at different time instances. This nonstationary nature of EEG limits the accuracy of it when subject independency is the priority. The aim of this study is to increase the subject-independent recognition accuracy by exploiting pretrained state-of-the-art Convolutional Neural Network (CNN) architectures. Unlike similar studies that extract spectral band power features from the EEG readings, raw EEG data is used in our study after applying windowing, pre-adjustments and normalization. Removing manual feature extraction from the training system overcomes the risk of eliminating hidden features in the raw data and helps leverage the deep neural network’s power in uncovering unknown features. To improve the classification accuracy further, a median filter is used to eliminate the false detections along a prediction interval of emotions. This method yields a mean cross-subject accuracy of 86.56% and 78.34% on the Shanghai Jiao Tong University Emotion EEG Dataset (SEED) for two and three emotion classes, respectively. It also yields a mean cross-subject accuracy of 72.81% on the Database for Emotion Analysis using Physiological Signals (DEAP) and 81.8% on the Loughborough University Multimodal Emotion Dataset (LUMED) for two emotion classes. Furthermore, the recognition model that has been trained using the SEED dataset was tested with the DEAP dataset, which yields a mean prediction accuracy of 58.1% across all subjects and emotion classes. Results show that in terms of classification accuracy, the proposed approach is superior to, or on par with, the reference subject-independent EEG emotion recognition studies identified in literature and has limited complexity due to the elimination of the need for feature extraction.

Emotion recognition (ER) is crucial for understanding human behaviours, social interactions, and psychological well-being. Electroencephalography (EEG) has emerged as a promising tool for capturing the neural correlates of emotions. This work is a systematic review of articles in ER using EEG signals. A total of 120 articles from 1041 articles were selected based on PRISMA guidelines using defined inclusion and exclusion criteria, published between 2018 and 2024. This article aims to provide an in-depth understanding of the current landscape of ER from EEG signals utilizing deep learning (DL). This review offers valuable guidance for researchers and practitioners seeking more refined and reliable emotion classification systems. To explore the effectiveness of DL models in EEG-based ER, several potential DL models, such as convolutional neural network, long short-term memory (LSTM), gated recurrent unit (GRU), hybrid bidirectional LSTM (BiLSTM), bidirectional GRU, and advanced DL models such as convolutional recurrent neural network and EEG-Conformer models are applied to two popular datasets, SEED and GAMEEMO, respectively, to depict the full process of ER. Additionally, the performance of DL models is also compared with the performance of basic machine learning (ML) models such as SVM, k-nearest neighbors, logistic regression, and boosting algorithms such as AdaBoost, XGBoost and LightGBM. Through extensive experiments and performance evaluations, the performance of different models when applied to the datasets mentioned above is compared. The accuracy, precision, recall, and F1-scores are analysed to determine the most effective model for EEG-based ER. The findings of this study demonstrate that the performance of hybrid DL models is more efficacious than that of ML models. The best-performing model (BiLSTM) classified the emotions, with an accuracy of 90.54% when applied to the GAMEEMO dataset. This research contributes to the growing body of literature on ER and provides insights into the feasibility of using EEG signals to understand emotional states, and presents a structured roadmap for future exploration. The findings can aid in the development of more accurate and reliable ER systems, which can have wide-ranging applications in psychology, social sciences, and human-computer interactions.

Atypical and subtle facial expression patterns in individuals with autism spectrum disorder (ASD) pose a significant challenge for automated emotion recognition. This study evaluates and compares the performance of convolutional neural networks (CNNs) and transformer-based deep learning models for facial emotion recognition in this population. Using a labeled dataset of emotional facial images, we assessed eight models across four emotion categories: natural, anger, fear, and joy. Our results demonstrate that transformer models consistently outperformed CNNs in both overall and emotion-specific metrics. Notably, the Swin Transformer achieved the highest performance, with an accuracy of 0.8000 and an F1-score of 0.7889, significantly surpassing all CNN counterparts. While CNNs failed to detect the fear class, transformer models showed a measurable capability in identifying complex emotions such as anger and fear, suggesting an enhanced ability to capture subtle facial cues. Analysis of the confusion matrix further confirmed the transformers’ superior classification balance and generalization. Despite these promising results, the study has limitations, including class imbalance and its reliance solely on facial imagery. Future work should explore multimodal emotion recognition, model interpretability, and personalization for real-world applications. Research also demonstrates the potential of transformer architectures in advancing inclusive, emotion-aware AI systems tailored for autistic individuals.

As the main expression medium of emotion, facial expression is used to explore personal emotion through facial expression recognition. The convolutional neural network architecture in deep learning can combine the emotion feature extraction and classification process, showing great advantages in emotion recognition by reducing the amount of computation and reference. In this paper, a deep learning-based facial expression recognition algorithm is proposed to realize expression analysis. The structure of multi-task convolutional neural network (MTCNN) is analyzed, the feature pyramid structure is introduced to design the C-MTCNN optimization algorithm, and the Inception v3 model is used to complete the feature extraction and emotion classification. The optimization of PCA algorithm and parameter adjustment on P-Net greatly improves the recognition accuracy of captured images, so as to realize the analysis of personal emotions. By testing the algorithm, the accuracy of the algorithm and the effect of the system are verified.

&lt;p&gt;&lt;span id="docs-internal-guid-c2be589c-7fff-55df-f7fc-a27d9748d2d4"&gt;&lt;span&gt;Human emotion recognition&lt;/span&gt;&lt;span&gt; is an upcoming research field of human computer interaction based on facial gestures and is being used for real-time analysis in classifying cognitive affective states from a facial video data. Since computers have become an integral part of life, many researchers are using emotion recognition and classification of data based on audio and text. But these approaches offer limited accuracy and relevance in emotion classification. Therefore we have introduced and analyzed a hybrid approach which could outperform the existing strategies that uses an innovative approach supported by selection of audio and video data characteristics for classification. The research uses SVM for classifying the data using audio-visual savee database and the results obtained show maximum classification accuracy with respect to audio data about 91.6 could be improved to 99.2% after the application of hybrid strategy. &lt;/span&gt;&lt;/span&gt;&lt;/p&gt;

Emotion recognition using artificial intelligence (AI) has become an increasingly vital area of research, offering transformative applications across healthcare, human-computer interaction, marketing, education, and entertainment. This paper provides a comprehensive review of the techniques and applications of AI in emotion recognition. Key methodologies, including facial expression analysis, speech emotion recognition, text-based emotion detection, physiological signal analysis, and multimodal approaches, are explored in detail. Each technique's underlying algorithms, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and natural language processing (NLP) methods, are examined for their efficacy in identifying human emotions. This paper also delves into practical applications, showcasing how emotion recognition enhances mental health monitoring, improves user experience, optimizes customer service, personalizes education, and enriches entertainment media. Moreover, this study addresses the ethical and technical challenges associated with emotion recognition, such as data privacy concerns, potential biases in AI models, and the need for accuracy and reliability in emotion detection systems. The discussion extends to future directions, emphasizing the integration of AI with wearable devices, advancements in multimodal systems, and the expansion of emotion recognition technology into new domains. By presenting a thorough analysis of current techniques and applications, this paper aims to highlight the significant impact and potential of AI-driven emotion recognition, paving the way for future innovations and ethical considerations in this dynamic field. The proliferation of Internet of Things (IoT) technology has enabled the development of smart home systems that can enhance home security, energy efficiency, and convenience. This paper presents the design and implementation of an IoT-based smart home security system that utilizes a variety of sensors and actuators to monitor and control various aspects of home security. The proposed system integrates multiple components, including motion detectors, door/window sensors, cameras, smart locks, and a central control unit, all interconnected through a wireless network. The system leverages cloud computing and machine learning techniques to process sensor data, detect potential threats, and initiate appropriate responses. A user-friendly mobile application allows homeowners to monitor and control the security system remotely. The paper discusses the system architecture, hardware and software components, communication protocols, and security considerations. The implemented system demonstrates the feasibility and potential benefits of IoT-based smart home security solutions.

-- Speech signals are being considered as most effective means of communication between human beings. Many researchers have found different methods or systems to identify emotions from speech signals. Here, the various features of speech are used to classify emotions. Features like pitch, tone, intensity are essential for classification. Large number of the datasets are available for speech emotion recognition. Firstly, the extraction of features from speech emotion is carried out and then another important part is classification of emotions based upon speech. Hence, different classifiers are used to classify emotions such as Happy, Sad, Anger, Surprise, Neutral, etc. Although, there are other approaches based on machine learning algorithms for identifying emotions. Speech Emotion Recognition is a current research topic because of its wide range of applications and it became a challenge in the field of speech processing too. We have carried out a brief study on Speech Emotion Analysis along with Emotion Recognition. Speech Emotion Recognition (SER) can be defined as extraction of the emotional state of the speaker from his or her speech signal. There are few universal emotions including Neutral, Anger, we have worked on different tools to be used in SER. SER is tough because emotions are subjective and annotating audio is challenging task. Emotion recognition is the part of speech recognition which is gaining more popularity and need for it increases enormously. We have classified based on different type of emotions to detect from speech. Key Words: Speech Emotion Recognition, Affective Computing, Machine Learning, Deep Learning, Audio Signal Processing, Emotion Classification, Feature Extraction, Prosodic Features, Spectral Features, Mel-Frequency Cepstral Coefficients (MFCCs), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), Attention Mechanisms, Multimodal Emotion Recognition, Speaker-Independent SER, Real-Time Emotion Detection, Noise-Robust Emotion Recognition, Data Augmentation, Emotion-Aware Applications

As one of the most important research fields in the brain-computer interface (BCI) field, electroencephalogram (EEG) classification has a wide range of application values. However, for the EEG signal, it is difficult for the traditional neural networks to capture the characteristics of the EEG signal more comprehensively from the time and space dimensions, which has a certain effect on the accuracy of EEG classification. To solve this problem, we can improve the accuracy of classification via end-to-end learning of the time and space dimensions of EEG. In this paper, a new type of EEG classification network, the separable EEGNet (S-EEGNet), is proposed based on Hilbert-Huang transform (HHT) and a separable convolutional neural network (CNN) with bilinear interpolation. The EEG signal is transformed into time-frequency representation by HHT, which allows the EEG signal to be better described in the frequency domain. Then, the depthwise and pointwise elements of the network are combined to extract the feature map. The displacement variable is added by the bilinear interpolation method to the convolution layer of the separable CNN, allowing the free deformation of the sampling grid. The deformation depends on the local, dense, and adaptive input characteristics of the EEG data. The network can learn from the time and space dimensions of EEG signals end to end to extract features to improve the accuracy of EEG classification. To show the effectiveness of S-EEGNet, the team used this method to test two different types of EEG public datasets (motor imagery classification and emotion classification). The accuracy of motor imagery classification is 77.9%, and the accuracy of emotion classification is 89.91%, and 88.31%, respectively. The experimental results showed that the classification accuracy of S-EEGNet improved by 3.6%, 1.15%, and 1.33%, respectively.

Speech represents a fundamental medium for conveying human emotions and, as a result, speech-based emotion recognition (SER) systems have become pivotal in advancing human-computer interaction (HCI) across a range of applications. While significant progress has been made in speech emotion recognition over recent years, existing solutions still face several key challenges, in that they: <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(i)$</tex-math></inline-formula> rely excessively on subjectively annotated (discrete) labels during training, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(ii)$</tex-math></inline-formula> often overlook the label ambiguity of speech samples that express more than one class of emotions, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(iii)$</tex-math></inline-formula> underutilize unlabeled or ambiguous speech, for which typically a label distribution (or so-called soft labels) is available. To address these issues, we propose in this paper a novel SER model that explicitly handles ambiguous speech samples and overcomes the shortcomings outlined above. Central to our approach is a novel real-time soft-label correction strategy designed to refine the annotations assigned to ambiguous speech. The proposed model leverages both, (explicitly) labeled as well as ambiguous samples and applies the dynamic soft-label correction strategy alongside an enhanced inter-class difference loss function to iteratively optimize the label distributions during training. We theoretically demonstrate that our method is capable of approximating the true emotional distribution of speech even in the presence of label noise, suggesting that utilizing ambiguous speech samples without explicit emotion labels still contributes toward more effective emotion recognition. Furthermore, we integrate the representational power of convolutional neural networks (CNNs) with the contextual modeling capabilities of Wav2Vec 2.0 to enable a comprehensive extraction of spatio-temporal speech features. Experimental results on the IEMOCAP multi-label dataset confirm the effectiveness of our approach, achieving state-of-the-art performance with significant improvements in weighted accuracy (WA) and unweighted accuracy (UA) over competing methods.

Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network-time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31-50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

The current work on emotion recognition considering EEG signals has attracted significant attention due to its advantages in fields like mental health, neural computing, and human-computer interaction. Nonetheless, accurately classifying emotions using EEG signals poses a challenge, mainly due to issues such as noise, artefacts, and high dimensionality found within EEG data. Moreover, the current problem lies in extracting meaningful emotional features while efficiently addressing the above-described challenges to improve classification accuracy. Hence, this work proposed an approach for solving the challenges in emotion classification by presenting a model that integrates Principal Component Analysis (PCA) and Convolutional Neural Network (CNN). The PCA was used for removing noise and for dimensionality reduction. In contrast, the CNN was used for extracting emotion-related features and classifying emotions, mainly the two classes, arousal and valence. The proposed PCA-CNN model was evaluated considering the DEAP dataset, where the PCA-CNN model achieved 93.58% accuracy for arousal and 92.75% for valence for subject-dependent and 81.40% accuracy for arousal and 79.84% for valence for subject-independent. The findings also show that the PCA-CNN model provided better results in handling noise and reducing data complexity, ensuring high classification performance. The novelty of the PCA-CNN model is that PCA was used for preprocessing the EEG data, and CNN was used for feature optimization, which created a more efficient emotion classifier.

Strong emotions can be expressed and evoked via music. However, accurately recognizing the emotions in music using computational models is extremely challenging. When the music portions convey various complex emotions, the problem’s difficulty can rise significantly. In this work, a systematic strategy that blends multiple cutting-edge techniques is used to focus on the emotional classification of music. The Kaggle-sourced music emotion dataset provides a wide range of audio samples illustrating various emotional expressions in music. Spectral subtraction as a noise reduction approach is used to improve the quality of the analysis. This efficiently reduced unwanted background noise and improved the audio signals’ clarity. A Quantum Convolutional Neural Network (QCNN) is used for feature extraction, taking advantage of its special capacity to extract complex patterns and characteristics from the music data that are essential for emotion recognition. To provide accurate predictions, the training procedure is optimized and finally utilized Flexible Runge Kutta Optimized Multilayer Perceptrons (FRKO-MLP) for emotion classification. The goal of this combination of advanced classification approaches, quantum feature extraction and noise reduction is to increase the robustness and accuracy of music emotion identification systems. Based on the chosen datasets, the results of the experimental study demonstrate that the suggested model greatly increases the efficiency and accuracy of emotion classification in music. The proposed FRKO-MLP attains 93.21% accuracy, 90.52% precision, 87.34% sensitivity and 89.72% F1-score. These findings show that the model is far better at identifying emotions than traditional methods.