Deep learning-based stress detection using encoded physiological signal representations

A stress-detection system is proposed based on physiological signals. Concretely, galvanic skin response (GSR) and heart rate (HR) are proposed to provide information on the state of mind of an individual, due to their nonintrusiveness and noninvasiveness. Furthermore, specific psychological experiments were designed to induce properly stress on individuals in order to acquire a database for training, validating, and testing the proposed system. Such system is based on fuzzy logic, and it described the behavior of an individual under stressing stimuli in terms of HR and GSR. The stress-detection accuracy obtained is 99.5% by acquiring HR and GSR during a period of 10 s, and what is more, rates over 90% of success are achieved by decreasing that acquisition period to 3-5 s. Finally, this paper comes up with a proposal that an accurate stress detection only requires two physiological signals, namely, HR and GSR, and the fact that the proposed stress-detection system is suitable for real-time applications.

As an essential challenge within the realm of affective computing, emotion recognition assumes a vital role in bestowing computers with a higher level and comprehensive intelligence. Furthermore, it has emerged as a crucial research topic in both human–computer interaction (HCI) and medical rehabilitation related to mental illnesses. However, the related fusion studies for modeling physiological signals in emotion recognition are less based on multimodal coordinated representation and lack the exploration of multimodal physiological signal correlation. In this article, we propose a tensor correlation fusion framework for emotion recognition based on multimodal physiological signals. After extracting effective features from various physiological signals, the coordinated representation module of the framework first simultaneously learns the linear correlation of all input physiological signals based on the covariance tensor. An optimized solution strategy is constructed to obtain the coordinated representation corresponding to each physiological signal. Finally, an emotion recognition module fuses the correlation information of the coordinated representation of different physiological signals as input to the emotion recognition classifier. This framework constructs coordinated representation by introducing a strategy to simultaneously capture the correlation among multiple physiological signals, providing a fresh perspective with a well-defined mathematical foundation for the fusion of multimodal physiological signals in the realm of emotion recognition. The experiments conducted on the DEAP dataset demonstrate that compared with related methods, the framework achieves relatively higher emotion recognition performance while obtaining a coordinated representation of multimodal physiological signal correlations of emotions, all while achieving superior processing speed.

Stress detection has become an increasingly important area of research, with applications spanning healthcare, occupational safety, and mental health. Machine learning (ML) algorithms have shown considerable potential in identifying stress from various data sources such as physiological signals, voice recordings, and even text. This review paper explores and compares the two primary categories of machine learning algorithms supervised and unsupervised for stress detection. We systematically review existing studies, discuss their methodologies, and analyze the strengths and weaknesses of each approach. The goal is to provide insights into the most effective methods for automatic stress detection and highlight the challenges and opportunities for future research.

Continuous and multimodal stress detection has been performed recently through wearable devices and machine learning algorithms. However, a well-known and important challenge of working on physiological signals recorded by conventional monitoring devices is missing data due to sensors insufficient contact and interference by other equipment. This challenge becomes more problematic when the user/patient is mentally or physically active or stressed because of more frequent conscious or subconscious movements. In this paper, we propose ReLearn, a robust machine learning framework for stress detection from biomarkers extracted from multimodal physiological signals. ReLearn effectively copes with missing data and outliers both at training and inference phases. ReLearn, composed of machine learning models for feature selection, outlier detection, data imputation, and classification, allows us to classify all samples, including those with missing values at inference. In particular, according to our experiments and stress database, while by discarding all missing data, as a simplistic yet common approach, no prediction can be made for 34% of the data at inference, our approach can achieve accurate predictions, as high as 78%, for missing samples. Also, our experiments show that the proposed framework obtains a cross-validation accuracy of 86.8% even if more than 50% of samples within the features are missing.

emotional states detection from physiological signals has many potential applications. In Human-Machine or Human-Human interaction systems, stress detection could provide users with improved services and can be a tool for monitoring and preventing potential stress-related pathologies. Traditional machine learning techniques for automatic stress recognition have been used in previous researches but they sometimes present specific limitations. The emergence of deep learning permits the reveal of underlying patterns in body response witch, otherwise would not be easily observed. In this paper we explore the application of Long Short-Term Memory (LSTM) and Deep Neural (DNN) Networks for real time stress monitoring in young gamers. We base our study on their body responses. For this, we use physiological signals such as the electrocardiography (ECG), the electrodermal activity (EDA), and the electromyography (EMG), measured by non-invasive wearable sensors. The result of the study provides an evaluation of both models' capacity in predicting real time gamers' emotional state built on the variation of their physiological parameters.

This paper presents a novel approach for mental stress detection. In proposed system, three signals including Pupil Diameter (PD), Electrocardiogram (ECG) and Photoplethysmogram (PPG) are analyzed using the soft computing techniques, and most relevant features are extracted from each one. Then, the optimized features are selected by using the Genetic Algorithm (GA) and imported into the Fuzzy SVM (FSVM) to classify "stress" and "relaxation" states. In order to evaluate the performance of proposed system, a multimodal dataset consisting of pupil video, ECG and PPG signals are constructed; a Stroop color-word (SCW) test is designed to act as the stimulus to induce stress in healthy subjects. The experimental results demonstrate the physiological signals have great potential for stress detection, and the proposed system provides high classification performance.

Rich feature extraction is essential to train a good machine learning (ML) framework. These features are generally extracted separately from each modality. We hypothesize that richer features can be learned when modalities are jointly explored. These joint modality features can perform better than those extracted from individual modalities. We study two modalities, physiological signals – Electrodermal activity (EDA) and electrocardiogram (ECG) to investigate this hypothesis. We investigate our hypothesis to achieve three objectives for subject-independent stress detection. For the first time in the literature, we apply our proposed framework in the frequency domain. The frequency-domain decomposition of the signal effectively separates it into periodic and aperiodic components.We can correlate their behaviour by focusing on each band of the signal spectrum. Second, we show that our framework outperforms late fusion, early fusion and other notable works in the field. Finally, we validate our approach on four benchmark datasets to show its generalization ability.

Aims: Extended reality (XR) has been widely recognized for its ability to evoke empathetic responses by immersing users in virtual scenarios and promoting perspective-taking. However, to fully realize the empathic potential of XR, it is necessary to move beyond the concept of XR as a unidirectional “empathy machine.” This study proposes a bidirectional “empathy-enabled XR” framework, wherein XR systems not only elicit empathy but also demonstrate empathetic behaviors by sensing, interpreting, and adapting to users’ affective and cognitive states. Methods: Two complementary frameworks are introduced. The first, the Empathic Large Language Model (EmLLM) framework, integrates multimodal user sensing (e.g., voice, facial expressions, physiological signals, and behavior) with large language models (LLMs) to enable bidirectional empathic communication. The second, the Matrix framework, leverages multimodal user and environmental inputs alongside multimodal LLMs to generate context-aware 3D objects within XR environments. This study presents the design and evaluation of two prototypes based on these frameworks: a physiology-driven EmLLM chatbot for stress management, and a Matrix-based mixed reality (MR) application that dynamically generates everyday 3D objects. Results: The EmLLM-based chatbot achieved 85% accuracy in stress detection, with participants reporting strong therapeutic alliance scores. In the Matrix framework, the use of a pre-generated 3D model repository significantly reduced graphics processing unit utilization and improved system responsiveness, enabling real-time scene augmentation on resource-constrained XR devices. Conclusion: By integrating EmLLM and Matrix, this research establishes a foundation for empathy-enabled XR systems that dynamically adapt to users’ needs, affective and cognitive states, and situational contexts through real-time 3D content generation. The findings demonstrate the potential of such systems in diverse applications, including mental health support and collaborative training, thereby opening new avenues for immersive, human-centered XR experiences.

This paper presents a fuzzy logic model for Real-time Stress Detection (RSD) that allows continuous and in-depth evaluation of user’s stress in a straightforward and systematic way by using physiological data. The RSD model takes as input signals’ values (galvanic skin response, heart rate) and then generates the corresponding level for user’s stress (low, mid-low, mid-high and high). To this end, skin conductance and heart rate signals were used to determine the stress levels classes. Results showed that the proposed tool-based stress detection mechanism can support a systematic evaluation of user’s stress in real time. Given that stress is highly subjective and may largely depend both on context and user characteristics, such results are rather encouraging for such a challenging problem. From a UX evaluators’ perspective, a preliminary study, involving three HCI experts, investigated the usefulness of the proposed RSD mechanism and revealed that it can substantially decrease time and effort required to make sense of user testing data. Furthermore, practitioners reported that using RSD they could conduct a more in-depth analysis compared to their current practices.

Mental stress can lead to traffic accidents by reducing a driver’s concentration or increasing fatigue while driving. In recent years, demand for methods to detect drivers’ stress in advance to prevent dangerous situations increased. Thus, we propose a novel method for detecting driving stress using nonlinear representations of short-term (30 s or less) physiological signals for multimodal convolutional neural networks (CNNs). Specifically, from hand/foot galvanic skin response (HGSR, FGSR) and heart rate (HR) short-term input signals, first, we generate corresponding two-dimensional nonlinear representations called continuous recurrence plots (Cont-RPs). Second, from the Cont-RPs, we use multimodal CNNs to automatically extract FGSR, HGSR, and HR signal representative features that can effectively differentiate between stressed and relaxed states. Lastly, we concatenate the three extracted features into one integrated representation vector, which we feed to a fully connected layer to perform classification. For the evaluation, we use a public stress dataset collected from actual driving environments. Experimental results show that the proposed method demonstrates superior performance for 30-s signals, with an overall accuracy of 95.67%, an approximately 2.5–3% improvement compared with that of previous works. Additionally, for 10-s signals, the proposed method achieves 92.33% classification accuracy, which is similar to or better than the performance of other methods using long-term signals (over 100 s).

Automatic stress detection using heart rate variability (HRV) features has gained significant traction as it utilizes unobtrusive wearable sensors measuring signals like electrocardiogram (ECG) or blood volume pulse (BVP). However, detecting stress through such physiological signals presents a considerable challenge owing to the variations in recorded signals influenced by factors, such as perceived stress intensity and measurement devices. Consequently, stress detection models developed on one dataset may perform poorly on unseen data collected under different conditions. To address this challenge, this study explores the generalizability of machine learning models trained on HRV features for binary stress detection. Our goal extends beyond evaluating generalization performance; we aim to identify the characteristics of datasets that have the most significant influence on generalizability. We leverage four publicly available stress datasets (WESAD, SWELL-KW, ForDigitStress, VerBIO) that vary in at least one of the characteristics such as stress elicitation techniques, stress intensity, and sensor devices. Employing a cross-dataset evaluation approach, we explore which of these characteristics strongly influence model generalizability. Our findings reveal a crucial factor affecting model generalizability: primary stressor. Models achieved good performance across datasets when the primary stressor (e.g., social evaluation in our case) remains consistent. Factors like stress intensity or brand of the measurement device had minimal impact on cross-dataset performance. Based on our findings, we recommend matching the primary stressor when deploying HRV-based stress models in new environments. Although previous works have performed cross-dataset evaluation of stress models, this is the first study to systematically investigate the factors influencing the cross-dataset applicability of HRV-based stress models. Our insights are crucial for scenarios with limited data, where techniques like domain generalization and domain adaptation may not be applicable.

Timely detection of an individual's stress level has the potential to improve stress management, thereby reducing the risk of adverse health consequences that may arise due to mismanagement of stress. Recent advances in wearable sensing have resulted in multiple approaches to detect and monitor stress with varying levels of accuracy. The most accurate methods, however, rely on clinical-grade sensors to measure physiological signals; they are often bulky, custom made, and expensive, hence limiting their adoption by researchers and the general public. In this article, we explore the viability of commercially available off-the-shelf sensors for stress monitoring. The idea is to be able to use cheap, nonclinical sensors to capture physiological signals and make inferences about the wearer's stress level based on that data. We describe a system involving a popular off-the-shelf heart rate monitor, the Polar H7; we evaluated our system with 26 participants in both a controlled lab setting with three well-validated stress-inducing stimuli and in free-living field conditions. Our analysis shows that using the off-the-shelf sensor alone, we were able to detect stressful events with an F1-score of up to 0.87 in the lab and 0.66 in the field, on par with clinical-grade sensors.

In this work, we classify the stress state of car drivers using multimodal physiological signals and regularized deep kernel learning. Using a driving simulator in a controlled environment, we acquire electrocardiography (ECG), electrodermal activity (EDA), photoplethysmography (PPG), and respiration rate (RESP) from N = 10 healthy drivers in experiments of 25min duration with different stress states (5min resting, 10min driving, 10min driving + answering cognitive questions). We manually remove unusable segments and approximately 4h of data remain. Multimodal time and frequency features are extracted and employed to regularized deep kernel machine learning based on a fusion framework. Task-specific representations of different physiological signals are combined using intermediate fusion. Subsequently, the fused multimodal features are fed a support vector machine (SVM) and a random forest (RF) for stress classification. The experimental results show that the proposed approach can discriminate between stress states. The combination of PPG and ECG using RF as classifier yields the highest F1-score of 0.97 in the test set. PPG only and RF yield a maximum F1-score of 0.90. Furthermore, subject-specific cross-validation improves performance. ECG and PPG signals are reliable in classifying the stress state of a car driver. In summary, the proposed framework could be extended to real-time stress state assessment in driving conditions.

Sleep stage classification is critical for diagnosing sleep quality. While deep neural networks are becoming popular for automatic sleep stage classification with supervised learning, large-scale labeled datasets are still hard to acquire. Recently, self-supervised learning (SSL) has become one of the most prominent approaches to alleviate the burden of labeling works. However, existing SSL methods are mainly designed for non-temporally correlated data and only learning representations from an instance level. Hence, the objective of this paper is to learn robust and generalizable representations for physiological signals with self-supervised learning. Specifically, we make the following contributions: (1) we propose a novel co-training scheme by exploiting complementary information from multiple views (time view and frequency view) of physiological signals to mine more positive samples, which overcomes the drawback of popular InfoNCE and achieves semantic-level representation learning; (2) we extend our framework with a memory module, implemented by a queue and a moving-averaged encoder, to enlarge the pool of negative candidates and keep the up-to-date representation; (3) extensive experiments conducted on sleep stage classification demonstrate state-of-the-art performance compared with SSL baselines, achieving 71.6% and 57.9% accuracies on two sleep datasets, SleepEDF and ISRUC respectively. The code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/larryshaw0079/CoSleep</uri> .

Stress, a common response to challenging situations, has become pervasive in contemporary daily life due to various factors. Persistent stress can weaken the human immune system, increasing the risk of chronic stress and contributing to a range of physical and mental health disorders. Therefore, timely detection of stress in its early stages is crucial for preventing adverse health outcomes. Physiological signals offer insights into the body's stress‐induced changes and can be leveraged for stress detection applications. Among these signals, the photoplethysmogram (PPG) signal stands out due to its advantages. This article introduces an innovative stress detection model based on dilated convolutional neural networks (Dilated CNNs), a deep learning algorithm. This model distinguishes between an individual's stressed and non‐stressed states by analysing PPG signals without requiring pre‐processing, denoising, or feature extraction. Leveraging the Empatica E4 PPG signals from the Wearable Stress and Affect Detection (WESAD) dataset, the authors developed and evaluated the model, achieving remarkable results: a test accuracy of 93.56% and an area under the curve (AUC) of 96.52%. These outcomes are particularly noteworthy given the streamlined data preparation process and methodological simplicity. Beyond enabling early stress diagnosis, this advancement holds promise for enhancing overall health and well‐being in the fast‐paced and intricate world. Additionally, its simplicity makes it suitable for real‐time stress detection and integration into wearable devices.