Edge4Emotion: An Edge Computing based Multi-source Emotion Recognition Platform for Human-Centric Software Engineering

Context-aware computing has become an essential component of modern wearable technology, enabling devices to sense, interpret, and respond to users’ environments and affective states. By integrating contextual intelligence, smart systems can adapt their own behavior to better align with user needs, enhancing efficiency, personalization, and user experience. A critical aspect of context awareness is the ability to recognize a user’s internal state, particularly emotions, which influence decision-making, communication, and overall well-being. While emotion recognition is traditionally approached through multimodal sensing—using cameras, microphones, or physiological sensors such as Electroencephalography (EEG), Electrocardiography (ECG), and Electro-dermal Activity (EDA)—these methods often require multiple sensors, leading to challenges related to power consumption, complexity, and privacy. The demand for a more efficient and unobtrusive approach to emotion and context recognition has driven research toward minimal-sensor solutions that can extract meaningful affective and contextual information with reduced hardware requirements. Restricting sensing sources of contextual information by using only limited number of sensors might come at the expense of constraining the ultimate goal of context-awareness. Such expense can be mitigated by “doing more with less” i.e. delving into the opportunities of using a single sensor for multiple purposes. What is desirable is a minimal-sensor solution capable of capturing both motion and physiological signals for both robust emotion recognition and additional context estimation. Among wearable sensors, chest-worn Inertial Measurement Units (IMUs) stand out as promising candidates for comprehensive context estimation, as they are well-positioned to capture motion patterns, and physiological vibrations, being the subtle cardio-respiratory signals. This raises the question of whether a single chest-worn IMU could serve as a minimal yet effective sensor for simultaneously estimating both emotional and contextual information, offering a practical alternative to traditional multi-sensor approaches. To address the challenge of using chest-worn IMU for multiple context sensing, I investigated the potential of using Seismocardiography (SCG) and Accelerometry-Derived Respiration (ADR) to estimate emotional states. First, I explored the existing capabilities of chest-worn IMUs in capturing cardio-respiratory vibrations and motion data, which I used as a knowledge base for embedding such IMUs in context sensing. Data plays a crucial role in this problem, therefore I developed a flexible software interface, ColEmo, to facilitate the scalable collection of emotion-related data and introduced a novel dataset, EmoWear, integrating physiological and motion data for emotion recognition and context awareness. Then, I examined the feasibility of using SCG for emotion recognition, comparing its performance with traditional physiological signals and evaluating its effectiveness as a single-sensor solution. This thesis presents a detailed analysis of the potential of chest-worn IMUs for emotion and context recognition. It reports on the design and validation of a SCG-based wearable system, the development and characteristics of the EmoWear dataset, and the implementation of the ColEmo platform for emotion data collection. The research also examines machine learning techniques applied to the SCG and ADR signals, assessing their effectiveness in estimating emotional states. My findings indicate that SCG-based emotion recognition achieves performance comparable to conventional physiological counterparts, demonstrating the feasibility of using a single IMU sensor for affective computing. The publicly-available EmoWear dataset provides a valuable resource for future research, highlighting the potential of combining motion and physiological data for enhanced context awareness. Additionally, the ColEmo software interface, which I published open-source, proves to be an effective tool for multi-modal emotion data collection, facilitating further advancements in wearable emotion recognition. The results pave the way for a minimal-sensor approach for context information estimation that can significantly reduce system complexity while maintaining high performance. These findings confirm that chest-worn IMUs present an opportunity to further improve emotion recognition in addition to the motion-based context information estimation. My research highlights the need for standardized methodologies in affective computing and demonstrates the viability of motion-based physiological sensing for emotion estimation. By using a single chest-worn IMU, this work presents an alternative to multi-sensor systems, reducing power consumption, and privacy concerns while maintaining robust performance. Future research should focus on refining IMU-based emotion recognition using deep learning techniques and larger datasets to enhance classification accuracy. Further validation in real-world settings is necessary to assess the practical applicability of the proposed system. The EmoWear dataset should be explored for additional context-awareness applications, such as activity recognition and stress monitoring, to maximize its utility. The ColEmo platform offers opportunities for expansion beyond emotion recognition, including cognitive load estimation, stress monitoring, and human-computer interaction. This research moves wearable technology forward by using fewer sensors, helping make affective computing more efficient, versatile, and respectful of privacy.

Human emotion recognition has emerged as a critical component in the development of intelligent systems capable of understanding and responding to human emotional states. This comprehensive review examines the latest novel deep learning methods for emotion recognition, encompassing advances in neural architectures, multimodal fusion techniques, and innovative approaches to feature extraction and classification. The field has witnessed remarkable progress through the integration of sophisticated deep learning models including convolutional neural networks (CNNs), long short-term memory networks (LSTMs), transformer architectures, and generative adversarial networks (GANs). Recent breakthroughs have demonstrated exceptional performance improvements, with some methods achieving over 99% accuracy on benchmark datasets such as DEAP, SEED, and MAHNOB-HCI. This review analyzes the evolution from traditional machine learning approaches to state-of-the-art deep learning methodologies, examining their effectiveness across various modalities including electroencephalography (EEG), facial expressions, speech, and physiological signals.The paper provides a systematic analysis of fusion strategies, architectural innovations, and performance metrics while identifying key challenges and future research directions in the rapidly evolving landscape of emotion recognition technology.

EmoCNN: Unleashing Human Emotions with Customized CNN Using Different Optimizers

IntroductionFacial expressions play a crucial role in human emotion recognition and social interaction. Prior research has highlighted the significance of the eyes and mouth in identifying emotions; however, limited studies have validated these claims using robust biometric evidence. This study investigates the prioritization of facial features during emotion recognition and introduces an optimized approach to landmark-based analysis, enhancing efficiency without compromising accuracy.MethodsA total of 30 participants were recruited to evaluate images depicting six emotions: anger, disgust, fear, neutrality, sadness, and happiness. Eye-tracking technology was utilized to record gaze patterns, identifying the specific facial regions participants focused on during emotion recognition. The collected data informed the development of a streamlined facial landmark model, reducing the complexity of traditional approaches while preserving essential information.ResultsThe findings confirmed a consistent prioritization of the eyes and mouth, with minimal attention allocated to other facial areas. Leveraging these insights, we designed a reduced landmark model that minimizes the conventional 68-point structure to just 24 critical points, maintaining recognition accuracy while significantly improving processing speed.DiscussionThe proposed model was evaluated using multiple classifiers, including Multi-Layer Perceptron (MLP), Random Decision Forest (RDF), and Support Vector Machine (SVM), demonstrating its robustness across various machine learning approaches. The optimized landmark selection reduces computational costs and enhances real-time emotion recognition applications. These results suggest that focusing on key facial features can improve the efficiency of biometric-based emotion recognition systems without sacrificing accuracy.

Computer recognition of human emotional states is an important component for efficient human-computer interaction. In this paper we explore an approach for recognition of human emotion from the visual information. We perform feature selection by using the two dimensions fractional Fourier transform. As a generalization of Fourier transform, the two dimensions fractional Fourier transform (2D-FrFT) contains simultaneity the time-frequency information of the signal, and is considered as a new tool for time-frequency analysis, especially in the area of image recognition. In this paper we use the amplitude of the 2D-FrFT of an image as the feature of the images to train the FLDA multiclassifier for human emotion recognition. Simulation result shows that this method of the features extraction and classifier has high recognition rate.

Emotion recognition using artificial intelligence (AI) has become an increasingly vital area of research, offering transformative applications across healthcare, human-computer interaction, marketing, education, and entertainment. This paper provides a comprehensive review of the techniques and applications of AI in emotion recognition. Key methodologies, including facial expression analysis, speech emotion recognition, text-based emotion detection, physiological signal analysis, and multimodal approaches, are explored in detail. Each technique's underlying algorithms, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and natural language processing (NLP) methods, are examined for their efficacy in identifying human emotions. This paper also delves into practical applications, showcasing how emotion recognition enhances mental health monitoring, improves user experience, optimizes customer service, personalizes education, and enriches entertainment media. Moreover, this study addresses the ethical and technical challenges associated with emotion recognition, such as data privacy concerns, potential biases in AI models, and the need for accuracy and reliability in emotion detection systems. The discussion extends to future directions, emphasizing the integration of AI with wearable devices, advancements in multimodal systems, and the expansion of emotion recognition technology into new domains. By presenting a thorough analysis of current techniques and applications, this paper aims to highlight the significant impact and potential of AI-driven emotion recognition, paving the way for future innovations and ethical considerations in this dynamic field. The proliferation of Internet of Things (IoT) technology has enabled the development of smart home systems that can enhance home security, energy efficiency, and convenience. This paper presents the design and implementation of an IoT-based smart home security system that utilizes a variety of sensors and actuators to monitor and control various aspects of home security. The proposed system integrates multiple components, including motion detectors, door/window sensors, cameras, smart locks, and a central control unit, all interconnected through a wireless network. The system leverages cloud computing and machine learning techniques to process sensor data, detect potential threats, and initiate appropriate responses. A user-friendly mobile application allows homeowners to monitor and control the security system remotely. The paper discusses the system architecture, hardware and software components, communication protocols, and security considerations. The implemented system demonstrates the feasibility and potential benefits of IoT-based smart home security solutions.

A dual framework for implicit and explicit emotion recognition: An ensemble of language models and computational linguistics

Speech signal processing is an active area of research, the most dominant source of exchanging information among human beings, and the best way for human–computer interaction (HCI). Human behavior assessments and emotion recognition from a speech signal, such as speech emotion recognition (SER) is an emerging HCI area of exploration with various real time claims. The performance of an efficient SER system depends on feature learning, which include salient and discriminative information such as high-level deep features. In this paper, we proposed a two-stream deep convolutional neural network with an iterative neighborhood component analysis (INCA) to learn mutually spatial-spectral features and select the most discriminative optimal features for the final prediction. Our model is composed of two channels, and each channel is associated with the convolutional neural network structure to extract cues from the oral signals. The first channel extracts feature from the spectral domain, and the second channel extracts features from the spatial domain, which are then fused and fed to the INCA to remove the severance and select the optimal features for the final model training. The joint refine features are passed from the fully connected network with a softmax classifier to yield the predictions of the different emotions. We trained our proposed system using three benchmarks, which included the EMO-DB, SAVEE, and RAVDESS emotional speech corpora, and we tested the prediction performance to secure 95%, 82%, and 85% recognition rates. The performance of the system shows the effectiveness and significance of the proposed system.

Making human–computer interaction more organic and personalized for users essentially demands advancement in human emotion recognition. Emotions are perceived by humans considering multiple factors such as facial expressions, voice tonality, and information context. Although significant research has been conducted in the area of unimodal/multimodal emotion recognition in videos using acoustic/visual features, few papers have explored the potential of textual information obtained from the video utterances. Humans experience emotions through their audio-visual and linguistic senses, making it quintessential to take the latter into account. This paper outlines two different algorithms for recognizing multimodal emotional expressions in online videos. In addition to acoustic (speech), visual (facial), and textual (utterances) feature extraction using BERT, we utilize bidirectional LSTMs to capture the context between utterances. To obtain richer sequential information, we also implement a multi-head self-attention mechanism. Our analysis utilizes the benchmarking CMU multimodal opinion sentiment and emotion intensity (CMU-MOSEI) dataset, which is the largest dataset for sentiment analysis and emotion recognition to date. Our experiments result in improved F1 scores in comparison to the baseline models.KeywordMultimodalEmotionRecognitionCMU-MOSEIMulti-head attentionBERTContext-aware

As an essential challenge within the realm of affective computing, emotion recognition assumes a vital role in bestowing computers with a higher level and comprehensive intelligence. Furthermore, it has emerged as a crucial research topic in both human–computer interaction (HCI) and medical rehabilitation related to mental illnesses. However, the related fusion studies for modeling physiological signals in emotion recognition are less based on multimodal coordinated representation and lack the exploration of multimodal physiological signal correlation. In this article, we propose a tensor correlation fusion framework for emotion recognition based on multimodal physiological signals. After extracting effective features from various physiological signals, the coordinated representation module of the framework first simultaneously learns the linear correlation of all input physiological signals based on the covariance tensor. An optimized solution strategy is constructed to obtain the coordinated representation corresponding to each physiological signal. Finally, an emotion recognition module fuses the correlation information of the coordinated representation of different physiological signals as input to the emotion recognition classifier. This framework constructs coordinated representation by introducing a strategy to simultaneously capture the correlation among multiple physiological signals, providing a fresh perspective with a well-defined mathematical foundation for the fusion of multimodal physiological signals in the realm of emotion recognition. The experiments conducted on the DEAP dataset demonstrate that compared with related methods, the framework achieves relatively higher emotion recognition performance while obtaining a coordinated representation of multimodal physiological signal correlations of emotions, all while achieving superior processing speed.

EEG signals are the modality that is widely used to recognize human emotions. However, the limited data on EEG signals remains challenging because of the small recording participants, the need for an expert to interpret EEG signals, and the expensive cost of tools to record EEG signals. This research proposed the data augmentation schemes on the EEG datasets to overcome the limited available data problem. Augmenting the data will help the generalizability of the emotion recognition model. The EEG signals on the DEAP and SEED datasets are transformed into image samples using a recurrence plot and spectrogram. Then, the artificial recurrence plot and the artificial spectrogram samples are generated using Pix2pix. This research used these artificial samples to conduct the data augmentation process. LeNet5, ResNet50, MobileNet, and DenseNet121 are used to conduct the classification. The best four data augmentation schemes are as follows: Appending 20,000 artificial recurrence plot samples to DEAP and SEED training datasets, appending 20,000 artificial spectrogram samples to the DEAP training dataset, and appending 15,000 artificial spectrogram samples to the SEED training dataset. The kappa coefficient for each classification model based on the best data augmentation schemes is computed. It is found that among the compared classifiers, LeNet5 achieved the best accuracy in both SEED (98.58%) and DEAP (86.12%) datasets when spectrogram was used. Therefore, LeNet5 trained on the spectrogram samples is a reliable and robust classification model. This finding implies that the use of spectrogram is more promising than the recurrence plot in human emotion recognition.

Detecting human emotion based on facial expression is considered a hard task for the computer vision community because of many challenges such as the difference of face shape from a person to another, difficulty of recognition of dynamic facial features, low quality of digital images, etc. In this paper, we propose a face-sensitive convolutional neural network (FS-CNN) for human emotion recognition. The proposed FS-CNN is used to detect faces on large scale images then analyzing face landmarks to predict expressions for emotion recognition. The FS-CNN is composed form two stages, patch cropping, and convolutional neural networks. The first stage is used to detect faces in high-resolution images and crop the face for further processing. The second stage is a convolutional neural network used to predict facial expression based on landmarks analytics, it was applied on pyramid images to process scale invariance. The proposed FS-CNN was trained and evaluated on the UMD Faces dataset. High performance was achieved with a mean average precision of about 95%.

Most of the electroencephalography (EEG) based emotion recognition systems rely on single stimulus to evoke emotions. EEG data is mostly recorded with higher number of electrodes that can lead to data redundancy and longer experimental setup time. The question “whether the configuration with lesser number of electrodes is common amongst different stimuli presentation paradigms” remains unanswered. There are publicly available datasets for EEG based human emotional states recognition. Since this work is focused towards classifying emotions while subjects are experiencing different stimuli, therefore we need to perform new experiments. Keeping aforementioned issues in consideration, this work presents a novel experimental study that records EEG data for three different human emotional states evoked with four different stimuli presentation paradigms. A methodology based on iterative Genetic Algorithm in combination with majority voting has been used to achieve configuration with reduced number of EEG electrodes keeping in consideration minimum loss of classification accuracy. The results obtained are comparable with recent studies. Stimulus independent configurations with lesser number of electrodes lead towards low computational complexity as well as reduced set up time for future EEG based smart systems for emotions recognition

Previous studies have shown that utilizing data sources such as historical or technical data along with deep learning approaches can increase the accuracy of forex price prediction. However, relying on a single data source, such as historical data alone, limits predictive performance as it fails to capture the entire complexity of the market. This study aims to demonstrate that merging financial data from four different sources can significantly improve prediction accuracy. This study proposes a hybrid LSTM-GRU-Attention deep learning model, leveraging historical, fundamental, technical, and sentiment data, to predict the closing price of the GBP/USD currency pair, which is actively traded on a global scale. QuantManager, Forexfactory, and DailyFX data were collected over three timeframes: 30-minute, 1-hour, and 1-day, from January 1, 2013 to December 31, 2023, respectively. The model’s performance was evaluated using MSE, RMSE, and MAE metrics. The proposed model with multi-source data demonstrated substantial error reductions compared to historical data only models from previous studies, achieving RMSE decreases from 17% to 89% and MAE reductions from 22% to 94% across 30-minute, 1-hour, and 1-day timeframes. The integrated multi-source data model outperformed models that integrated only historical data, across all timeframes, highlighting the benefits of improved forex prediction accuracy.

Nowadays, there is a high attention paid to human emotion recognition by using the electroencephalogram (EEG) signals based on machine learning technology. However, because of the high emotional variation, emotion recognition based on EEG is considered a high challenge in terms of pattern recognition. On the other hand, the Honey Badger (HB) algorithm is used to recognize several emotions such as happy, sad, angry, and others. Thus, this algorithm could attain several functions such as optimal selection of features, selection of a classifier, and setting of parameters based on various emotion datasets. Therefore, this paper presents a new method for human emotion recognition based on EEG signals. In the proposed method, the signals of the EEG are collected from the DREAMER database. Furthermore, the features of EEG signals are extracted using the Wavelet Transform (WT). The HB algorithm is used to select the most effective features of EEG signals and feed them to the classifier. For the classification part, there are several classifiers of Deep Neural Networks (DNNs) are used to classify the emotion classes. Experimental results show that the proposed method can achieve encouraging results with an accuracy of 82.5%. The proposed method using the HB algorithm demonstrates better performance than the standard features for analyzing EEG.