The field of affective computing (AffC) is a hot research topic, where keeping track of the latest state-of-theart can be cumbersome. Probably, due to this, a huge increase in publications of systematic reviews or surveys (SRoS) is appearing in different journals, covering various aspects such as databases, methods, and overall perspectives. Nevertheless, this increase does not mean more and better information, or at least a clarification of information. The present study analyses 10 SRoS, all published within the last 4 years, focusing only on covering AffC databases, with emphasis on collections where emotion or sentiment can be extracted from the body. It was observed that, depending on the SRoS, different information was presented, sometimes with missing or discrepant data, due to lack of information or by the way it was interpreted. As a result, from those 10 SRoS, a total of 111 different databases were analyzed, which were segmented into three groups (tiers, i.e., citation-based categorization) by their relative importance of appearance in the SRoS. In addition, it is proposed a taxonomy with a minimum set of characterizing information that researchers should address when publishing or reviewing databases.

We propose a novel architecture for agents to be employed in Human-AI Teaming in various, even critical, domains. The architecture is based on affective computing, empathy, and Theory of Mind (ToM) and uses a description of the user profile and the operational, professional, and ethical requirements of the domain in which the agent operates. Moreover, it encompasses: a Knowledge Graph (KG), which is aimed at representing the various kinds of knowledge of the scenario at hand; a Behavior Tree (BT) enhanced with empathy-aware capabilities; a neural component, which elaborates sensor input from devices that monitor the user and input from the KG. The enhanced BT is the component that interacts with the user, making actions or providing suggestions. It also returns feedback to the KG to update it based on user experience. This two-way interaction between KG and BT is a major novelty of our work. As further contributions, we showcase an application of the proposed architecture on a driver co-pilot scenario and provide an early stage implementation of our architecture, mainly based on a Prolog implementation of the BT component.

Electroencephalography (EEG) is an objective tool for emotion recognition with promising applications. However, the scarcity of labeled data remains a major challenge in this field, limiting the widespread use of EEG-based emotion recognition. In this paper, a semi-supervised <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</b>ual-stream <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</b>elf-attentive <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b>dversarial <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G</b>raph <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</b>ontrastive learning framework (termed as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DS-AGC</i>) is proposed to tackle the challenge of limited labeled data in cross-subject EEG-based emotion recognition. The DS-AGC framework includes two parallel streams for extracting non-structural and structural EEG features. The non-structural stream incorporates a semi-supervised multi-domain adaptation method to alleviate distribution discrepancy among labeled source domain, unlabeled source domain, and unknown target domain. The structural stream develops a graph contrastive learning method to extract effective graph-based feature representation from multiple EEG channels in a semi-supervised manner. Further, a self-attentive fusion module is developed for feature fusion, sample selection, and emotion recognition, which highlights EEG features more relevant to emotions and data samples in the labeled source domain that are closer to the target domain. Extensive experiments are conducted on four benchmark databases (SEED, SEED-IV, SEED-V, and FACED) using a semi-supervised cross-subject leave-one-subject-out cross-validation evaluation protocol. The results show that the proposed model outperforms existing methods under different incomplete label conditions with an average improvement of 2.17%, which demonstrates its effectiveness in addressing the label scarcity problem in cross-subject EEG-based emotion recognition.

Physiological signal-based emotion recognition requires effective fusion of multi-modal physiological signals to improve recognition accuracy. In this paper, a multi-kernel embedding fusion framework (MKEFF) is proposed for multi-modal physiological signal emotion recognition. Specifically, multi-kernel learning and kernel approximation techniques are used to compute the multi-kernel embeddings of the original feature vectors of each modality independently. The embeddings are then fed in parallel to their respective representation learning layer, where the proposed sparse relation learning method is applied to all the modalities to explore the correlation and diversity among them. Finally, a distribution alignment based fusion method is proposed to align each modality in the subspace, and a weighted summation fusion is performed to obtain the fused representations. Extensive cross-subject emotion recognition experiments are conducted on three public datasets, DEAP, DECAF, and SEED-IV, to evaluate the proposed method. The experimental results demonstrate that the proposed method achieves better classification performance and interpretability than the state-of-the-art methods.

The Affective Audio Dataset (AAD) is a new and novel dataset of non-musical, non-anthropomorphic sounds intended for use in affective research. Sounds are annotated for their affective qualities by sets of human participants. The dataset was created in response to a lack of suitable datasets within the domain of audio emotion recognition. A total of 780 sounds are selected from the BBC Sounds Library. Participants are recruited online and asked to rate a subset of sounds based on how they make them feel. Each sound is rated for arousal and valence. It was found that while evenly distributed, there was bias towards the low-valence, high-arousal quadrant, and displayed a greater range of ratings in comparison to others. The AAD is compared with existing datasets to check its consistency and validity, with differences in data collection methods and intended use-cases highlighted. Using a subset of the data, the online ratings were validated against an in-person data collection experiment with findings strongly correlating. The AAD is used to train a basic affect-prediction model and results are discussed. Uses of this dataset include, human-emotion research, cultural studies, other affect-based research, and industry use such as audio post-production, gaming, and user-interface design.

Relieving stress is crucial for the health of modern individuals. Deep breathing exercises have been shown to be effective for stress relief. However, the effectiveness of deep breathing exercises is limited if feedback on breathing patterns is not provided instantly. For this, we propose a wearable device called Anti-Stress, which can provide users with breathing patterns in real-time, enabling them to dynamically adjust their breathing according to Anti-Stress instructions for optimal relaxation. In addition, Anti-Stress provides users with objective stress indices by measuring the user's heart rate variability (HRV). The accuracy of Anti-Stress in detecting breathing patterns is up to 99%, and its response time is less than 200 ms, ensuring users can receive immediate guidance. To evaluate the effectiveness of Anti-Stress, we recruited 60 participants to conduct an empirical experiment. The ANCOVA analysis shows that the experimental group significantly reduces stress by using Anti-Stress compared to the control group without it. Furthermore, the usability questionnaires including SUS and QUIS demonstrate Anti-Stress's higher usability compared to a traditional non-biofeedback app. Our analysis also reveals that people with high openness or neuroticism personalities had better stress relief by using Anti-Stress. The results of this study demonstrate the feasibility and practicality of our respiratory feedback mechanism in helping users relieve stress and enhancing the effectiveness of deep breathing exercises.

Emotional stress significantly impacts mental and physical health, motivating the need for computational methods that can model subtle affective states. This study introduces a novel multimodal multitask architecture for the simultaneous regression of arousal and valence, two key emotional dimensions correlated with emotional stress. Utilizing the ULM-TSST dataset, the proposed model integrates video, audio, text, and physiological data through a combination of LSTM and Transformer Encoder networks, employing class tokens for task-specific representations. Experimental results demonstrate the model's effectiveness, achieving an average Concordance Correlation Coefficient (CCC) of 60% for valence and 61% for arousal, outperforming existing approaches by a 4% CCC margin. Ablation studies highlight the importance of each modality, confirming that the best performance is achieved only when all modalities are included. Additionally, comparative analysis between the multitask and single-task versions of the architecture confirms that the multitask approach outperforms single-task models in both arousal and valence prediction. This improvement underscores the benefits of shared representations and joint learning of related affective dimensions within a unified framework. The code for this project is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/cosbidev/Temporal-Multimodal-Multitask-Attention</uri>.

Facial Expression Recognition (FER) models based on deep learning mostly rely on a supervised train-once-test-all approach. These approaches assume that a model trained on an in-the-wild facial expression dataset with one type of domain distribution will perform well on a test dataset with a domain distribution shift. However, facial images in real-world can be from different domain distributions from which the model has been trained. However, re-training models on only new domain distributions will severely affect the performance of the previous domain. Re-training on all previous and new data can improve overall performance but is computationally expansive. In this study, we oppose the train-once-test-all approach and propose a buffer-based continual learning approach to enhance the performance of multiple in-the-wild datasets. We propose a model that continually leverages attention to important facial features from the pre-trained model to improve performance in multiple datasets. We validated our model using split-in-the-wild datasets where the dataset is provided to the model in an incremental setting instead of all at once. Furthermore, to evaluate the model performance, we continually used three in-the-wild datasets representing different domains (Domain-FER). Extensive experiments on these datasets reveal that the proposed model achieves better results than other Continual FER models.

Humans and robots are expected to collaborate closely sharing the same working environment. However, a quantitative measure of the perceived sense of control (SoC) in interacting with artificial systems is still an unmet need. This work introduces a ground-breaking approach that will profoundly impact the way of evaluating the SoC in human-robot interaction. We investigated the human behaviour during human-robot interactions by examining the Hand Blink Reflex (HBR). The HBR is an innate defensive reflex elicited by the electrical stimulation of the median nerve and can be measured using EMG recordings from the orbicularis oculi muscles. We recorded HBR in twenty subjects during different experimental conditions considering a robotic arm entering the defensive peripersonal space (DPPS) at near or far proximities to the face and under different control modalities: autonomous robot movement, human control limited to starting or stopping the robot, and human fully controlling the speed of the robot. According to predictions, the HBR amplitude is modulated by the proximity of the robotic arm to the DPPS. Crucially, the more the human confidence in the robot control, the lower the HBR amplitude. This novel method quantifies human confidence in robot control, potentially advancing human-robot collaboration by enhancing our understanding of the neural mechanisms underlying perceived control and safety in shared workspaces. Results can be further exploited for comparing the effectiveness of robot control interfaces and algorithms