A hybrid transposed attention based deep learning model for wearable and explainable stress recognition

Enhancing user experience and performance, including task load in immersive environments, requires accurate prediction of user gaze point, reaction time, and mental and physical load uptake. Current gaze prediction approaches focus primarily on motion-based information, lacking physiological data, which leads to poor prediction accuracy in highly dynamic virtual reality (VR) environments. Traditional cognitive load measurements rely on post-task analysis without proper multimodal data integration and fail to capture the real-time dynamics of user states during interaction. Likewise, reaction time or attention load are often assessed only after the interaction, without using real-time immersive sensor data, which limits adaptive responsiveness. To tackle these limitations, we leveraged a comprehensive multimodal dataset - VRWalking, which recorded timestamped eye-tracking metrics, physiological signals (heart rate and galvanic skin response), and behavioral performance data during real-time engagement in a VR environment. We developed a unified multitask model based on the MultiPatchFormer architecture, which processes multimodal VR signals through dual patch projection branches for gaze and classification inputs. The model employs multiscale patch embeddings, cross-attention between gaze and classification pathways, channel attention, and transformer encoders to jointly predict continuous user gaze and classify reaction time, cognitive load (mental load and physical load). Our methodology achieved excellent predictive performance: 95.64% for reaction time, 98.01% for mental load, and 97.45% for physical load, with a MAPE (Mean Absolute Percentage Error) of 15.24% for gaze prediction. We applied Shapley Additive explanations (SHAP) analysis to interpret the model’s behavior across all features, including eye-tracking, head-tracking, and physiological signals. The analysis revealed which features most influenced the predictions of user gaze, reaction time, mental load, and physical load. Our methods, while based only on the VRWalking dataset, demonstrated strong performance across all tasks, suggesting promising potential for real-world VR applications such as interactive training systems that respond to user attention lapses, educational platforms that adapt to cognitive load, and performance assessments that consider physiological indicators.

Current decoding algorithms based on a one-dimensional (1D) convolutional neural network (CNN) have shown effectiveness in the automatic recognition of emotional tasks using physiological signals. However, these recognition models usually take a single modal of physiological signal as input, and the inter-correlates between different modalities of physiological signals are completely ignored, which could be an important source of information for emotion recognition. Therefore, a complete end-to-end multi-input deep convolutional neural network (MI-DCNN) structure was designed in this study. The newly designed 1D-CNN structure can take full advantage of multi-modal physiological signals and automatically complete the process from feature extraction to emotion classification simultaneously. To evaluate the effectiveness of the proposed model, we designed an emotion elicitation experiment and collected a total of 52 participants' physiological signals including electrocardiography (ECG), electrodermal activity (EDA), and respiratory activity (RSP) while watching emotion elicitation videos. Subsequently, traditional machine learning methods were applied as baseline comparisons; for arousal, the baseline accuracy and f1-score of our dataset were 62.9 ± 0.9% and 0.628 ± 0.01, respectively; for valence, the baseline accuracy and f1-score of our dataset were 60.3 ± 0.8% and 0.600 ± 0.01, respectively. Differences between the MI-DCNN and single-input DCNN were also compared, and the proposed method was verified on two public datasets (DEAP and DREAMER) as well as our dataset. The computing results in our dataset showed a significant improvement in both tasks compared to traditional machine learning methods (t-test, arousal: p = 9.7E-03 < 0.01, valence: 6.5E-03 < 0.01), which demonstrated the strength of introducing a multi-input convolutional neural network for emotion recognition based on multi-modal physiological signals.

Behavioral factors can affect the blood glucose (BG) levels in people with type 1 diabetes (T1D), therefore, their effects need to be incorporated in blood glucose management for these individuals. Accordingly, in this work, we study the effect of two behavioral states, physical activity (PA) and stress state (SS), on BG fluctuations in individuals with T1D. We provide two methods for quantifying biomarkers related to PA and SS using raw acceleration (ACC) and electrodermal activity (EDA) data collected with a wearable device. We evaluate the impact of PA and SS on BG fluctuation by adding the derived behavior-related biomarkers in two cutting-edge deep learning-based glucose predictive models, a long short-term memory (LSTM) and a convolutional neural network (CNN)-LSTM network, for prediction horizons (PHs) of 30, 60 and 90 min. Through an ablation study, we demonstrate that incorporating the estimated behavior-related biomarkers improves the BG predictive model’s performance obtaining mean absolute error (MAE) 9.13 ± 0.95, 17.75 ± 1.93 and 31.85 ± 2.88 in [mg/dL], root mean square error (RMSE), 12.35 ± 1.06, 24.71 ± 2.31 and 41.64 ± 4.12 in [mg/dL], and coefficient of determination (R2), 95.34 ± 3.34, 78.87 ± 4.35 and 60.11 ± 4.76 in [%], for the LSTM model; and MAE 9.37 ± 0.88, 17.87 ± 1.67 and 29.47 ± 2.13 in [mg/dL], RMSE 12.51 ± 1.40, 24.37 ± 2.49 and 39.52 ± 3.89 in [mg/dL], and R2 94.65 ± 3.90, 78.37 ± 4.11 and 61.12 ± 4.30 in [%], for the CNN-LSTM model, respectively, across all PHs. Additionally, we illustrate the generalizability of the proposed models by performing both population- and patient-wise.

Pilot stress detection is a challenging task and it plays a vital role in improving flight performance and avoiding catastrophic accidents. Many deep learning models have been adopted for stress recognition. However, these models tend to ignore the dependencies between multimodal physiological signals, which can boost the model performance potentially. A transformer-based deep learning framework, which can obtain the position information of multimodal signals by combining a transformer network with a traditional convolutional neural network (CNN), is proposed for detecting pilot stress. The 14 pilots' physiological data, including electrocardiography (ECG), electromyography (EMG), electrodermal (EDA), respiration (RESP), and skin temperature (SKT), under different stress states are collected for training and validation, and evaluated among different state-of-the-art models. The results show that the proposed model achieves an accuracy of 93.28%, 88.75%, and 84.85% for two-, three-, and four-class classification tasks, respectively, showing faster integration and promising performance.

The drowsiness of pilot causes the various aviation accidents such as an aircraft crash, breaking away airline, and passenger safety. Therefore, detecting the pilot’s drowsiness is one of the critical issues to prevent huge aircraft accidents and to predict pilot’s mental states. Conventional studies have been investigated using physiological signals such as brain signals, electrodermal activity (EDA), electrocardiogram (ECG), respiration (RESP) for detecting pilot’s drowsiness. However, these studies have not sufficient performance to prevent sudden aviation accidents yet because it could detect the mental states after drowsiness occurred and only focus on whether drowsiness or not. To overcome the limitations, in this paper, we propose a multimodal convolutional bidirectional LSTM network (MCBLN) to detect drowsiness or not as well as drowsiness level using the fused physiological signals (electroencephalography (EEG), EDA, ECG, and RESP) for the pilot’s environment. We acquired the physiological signals for the pilot’s simulated aircraft environment across seven participants. The proposed MCBLN extracted the features considering the spatial-temporal correlation of between EEG signals and peripheral physiological measures (PPMs) (EDA, ECG, RESP) to detect the current pilot’s drowsiness level. Our proposed method achieved the grand-averaged 45.16% (±1.01) classification accuracy for 9-level of drowsiness. Also, we obtained 84.41% (±1.34) classification accuracy for whether the drowsiness or not across all participants. Hence, we have demonstrated the possibility of the not only drowsiness detection but also 9-level of drowsiness for the pilot’s aircraft environment.

Text classification is a classic task in the field of natural language processing. however, the existing methods of text classification tasks still need to be improved because of the complex abstraction of text semantic information and the strong relecvance of context. In this paper, we combine the advantages of two traditional neural network model, Long Short-Term Memory(LSTM) and Convolutional Neural Network(CNN). LSTM can effectively preserve the characteristics of historical information in long text sequences, and extract local features of text by using the structure of CNN. We proposes a hybrid model of LSTM and CNN, construct CNN model on the top of LSTM, the text feature vector output from LSTM is further extracted by CNN structure. The performance of the hybrid model is compared with that of other models in the experiment. The experimental results show that the hybrid model can effectively improve the accuracy of text classification.

Automatic artifact recognition and correction for electrodermal activity based on LSTM-CNN models

This paper explores a novel method for anxiety detection in older adults using simple wristband sensors such as electrodermal activity (EDA) and photoplethysmogram (PPG) and a context-based feature. The proposed method for anxiety detection combines features from a single physiological signal with an experimental context-based feature to improve the performance of the anxiety detection model. The experimental data for this work are obtained from a year-long experiment on 41 healthy older adults (26 females and 15 males) in the age range 60–80 with mean age \(73.36 \pm 5.25\) during a Trier Social Stress Test (TSST) protocol. The anxiety level ground truth was obtained from State–Trait Anxiety Inventory (STAI), which is regarded as the gold standard to measure perceived anxiety. EDA and blood volume pulse (BVP) signals were recorded using a wrist-worn EDA and PPG sensor, respectively. 47 features were computed from EDA and BVP signal, out of which a final set of 24 significantly correlated features were selected for analysis. The phases of the experimental study are encoded as unique integers to generate the context feature vector. A combination of features from a single sensor with the context feature vector is used for training a machine learning model to distinguish between anxious and not-anxious states. Results and analysis showed that the EDA and BVP machine learning models that combined the context feature along with the physiological features achieved 3.37% and 6.41% higher accuracy, respectively, than the models that used only physiological features. Further, end-to-end processing of EDA and BVP signals was simulated for real-time anxiety level detection. This work demonstrates the practicality of the proposed anxiety detection method in facilitating long-term monitoring of anxiety in older adults using low-cost consumer devices.

In the era of rapid e-commerce and social networks, the recognition of textual emotion has garnered escalating interest in recent years. This area of study proves pivotal for applications such as content recommendation and human-robot interaction. Prominent models such as CNN, LSTM, and Roberta have been used in natural language processing to decipher the subtleties of textual emotion. Despite the commendable accuracy achieved by Roberta, its extensive model parameters come at the cost of sluggish training speeds. To address this issue, we propose a novel hybrid approach that integrates Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks. We employ this hybrid model to address the emotion classification problem using a dataset named "Emotion", and we compare its performance with that of individual models like Roberta, CNN, and LSTM. The new hybrid model exhibits a more compact architecture while maintaining a relatively high accuracy rate, boasting an impressive 91%. This hybrid architecture can provide a good compromise between efficiency and performance. This research contributes to the ongoing exploration of efficient and accurate models for textual emotion recognition, emphasizing the significance of balancing model complexity and training speed in practical applications.

This study addresses the question whether galvanic skin response (GSR) and blood volume pulse (BVP) of untrained and unaided observers can be used to identify real and posed smiles from different sets of smile videos or smile images. Observers were shown smile face videos/images, either singly or paired, with the intention to recognise each viewed as real or posed smiles. We created four experimental situations, namely single images (SI), single videos (SV), paired images (PI), and paired videos (PV). The GSR and BVP signals were recorded and processed. Our machine learning classifiers reached the highest accuracy of 93.3%, 87.6%, 92.0%, 91.7% for PV, PI, SV, and SI respectively. Finally, PV and SI were found to be the easiest and hardest way to identify real and posed smiles respectively. Overall, we demonstrated that observers’ subconscious physiological signals (GSR and BVP) are able to identify real and posed smiles at a good accuracy.KeywordsPhysiological signalsAffective computingClassificationMachine learningSmile

Fibromyalgia is a musculoskeletal disorder characterized by chronic, widespread muscle pain. This condition is associated with disturbed sleep, which has a direct impact on patient quality of life. Patient-reported outcomes are frequently used to assess sleep quality, but show modest correlations with objective measures of sleep, such as polysomnography. Working towards our goal of an automated ambulatory system of assessing sleep quality, we use features from blood volume pulse (BVP) and electrodermal activity (EDA) collected with a wearable device during sleep. We compare these measurements between individuals with fibromyalgia who experienced poor sleep and individuals in a control group who experienced refreshing sleep. By applying Learning Using Concave and Convex Kernels (LUCCK) and Support Vector Machines (SVM), we achieve mean Area Under the Receiver Operating Characteristic Curve (AUC) of 0.6573 and 0.6526, respectively, by using BVP data for classifying individuals to the two groups.

Text classification is one of the basic tasks of natural language processing. In recent years, deep learning has been widely used in text classification tasks. The representative one is the convolutional neural network. The convolutional neural network(CNN) is limited by the size of the local window and can only extract local features of the text. For long texts like news, CNN cannot learn the longterm dependence of the long text. Another model of deep learning recurrent neural networks based on long short-term memory (LSTM) can learn the long-term dependence of text. Therefore, in the work of this paper, combining the advantages of CNN and LSTM, a LSTM_CNN Hybrid model is constructed for Chinese news text classification tasks. We first use LSTM to learn the longterm dependence of text, then we design a shallow convolution structure to further extract the semantic features of the text, and finally use the max-pooling operation to filter to obtain important features for classification. The model we proposed has achieved good results on the News dataset.

Mood-aware visual question answering

Stress impairs human cognition and over long periods of time can cause severe negative health effects for older adults. Real-time stress detection can be a useful tool in stress mitigation strategies, but stress detection is often implemented with machine learning models too large to run on an edge device. This paper presents a stress detection framework for older adults which is designed to be executed with low latency by a wearable device or other embedded system. We use Galvanic Skin Response, Blood Volume Pulse, Interbeat Interval, and skin temperature data with cortisol as a stress biomarker to evaluate the performance of several machine learning models. The best performing model was a radial basis function-based SVC (Support Vector Machine). The model achieved a test data accuracy of 82&#x0025; and a test data F1-Score of 0.70. Performance data indicates that the framework performs well and could be improved for use on a wearable device.

Human–robot collaboration (HRC) is a key focus of Industry 5.0, aiming to enhance worker productivity while ensuring well-being. The ability to perceive human psycho-physical states, such as stress and cognitive load, is crucial for adaptive and human-aware robotics. This paper introduces MultiPhysio-HRC, a multimodal dataset containing physiological, audio, and facial data collected during real-world HRC scenarios. The dataset includes electroencephalography (EEG), electrocardiography (ECG), electrodermal activity (EDA), respiration (RESP), electromyography (EMG), voice recordings, and facial action units. The dataset integrates controlled cognitive tasks, immersive virtual reality experiences, and industrial disassembly activities performed manually and with robotic assistance, to capture a holistic view of the participants’ mental states. Rich ground truth annotations were obtained using validated psychological self-assessment questionnaires. Baseline models were evaluated for stress and cognitive load classification, demonstrating the dataset’s potential for affective computing and human-aware robotics research. MultiPhysio-HRC is publicly available to support research in human-centered automation, workplace well-being, and intelligent robotic systems.