De-Identified Feature-based Visualization of Facial Expression for Enhanced Text Chat

This study examined the association of the experience of emotion and somatic changes. The study compared reported somatic changes generally experienced when anxious with the actual association of the experience of emotion and somatic changes as measured during a specific event. Emotions were measured as both general negative emotion as well as specific emotions: anger, disgust, fear, sadness and shame. Participants were volunteers from a Victorian university who agreed to watch a video depicting the dramatisation of child abuse. Throughout the video, participants indicated their experience of emotion. Measures were also taken throughout the procedure of facial expression and Galvanic Skin Response (GSR). In order to examine emotional-congruence, subjects were divided into three groups. These groups were divided according to the congruence of subjects' experienced emotion with autonomic changes and facial expressivity. Groups were divided separately for each of the emotion types. Where there was little difference between the reported experience of emotion and that, which would have been expected from the observed somatic changes, the subject was deemed to be in the Congruent Group. Subjects whose reported experience of emotion was greater or less than would be expected from observed somatic changes were allocated to the Over-reporter and Under-Reporter Groups respectively. This data was then compared to participants' reports of the number of somatic symptoms usually experienced when anxious. It was found that participants who under-report the experience of general negative-emotion compared with their observed somatic changes (both GSR and facial expressivity) had lower trait-somatic-anxiety (reported fewer somatic symptoms usually experienced when anxious). There was no significant difference between the Congruent Group and Over-Reporter Group.;The Under-Reporter Groups had significantly lower trait-somatic-anxiety than the Congruent Group when emotional-congruence was defined by fear and GSR, anger and GSR and sadness and facial expressivity. The actual association of shame and disgust with either somatic change, sadness with autonomic change and anger and fear with facial expressivity was unrelated to the number of somatic symptoms reported to be usually experienced when anxious. The results supported the idea that subjective reports of the number of somatic symptoms reported to be usually experienced when anxious reflect the actual association of somatic change and experience, but with limitations. The actual association of experience of fear with autonomic change seems to reflect the number of somatic symptoms reported to be usually experienced when anxious more than other emotions. Further for those for whom the experience of anger and negative-emotion has a greater association with somatic change, there was a greater number of somatic symptoms reported to be usually experienced when anxious. This would suggest that some people have a greater association of some experiences of emotion and somatic change. Furthermore, while there is an association between reported somatic changes generally experienced when anxious with the actual association of the experience of emotion and somatic changes as measured during a specific event, this was dependant on the association of the emotion types rather than being generalised for all emotions.

Automatic analysis of human facial expressions is one of the challenging problems in machine vision systems. It has many applications in human-computer interactions such as, social signal processing, social robots, deceit detection, interactive video and behavior monitoring. In this paper, we develop a new method for automatic facial expression recognition based on facial muscle anatomy and human face structure. The algorithm finds approximate location of effective facial muscles and extracts features by measuring skin texture in 11 local patches. Seven facial expressions, including neutral are being classified in this study using AdaBoost classifier and other classifiers on MMI databases. Experimental results show that analyzing skin texture from selected local patches gives accurate and efficient information in order to identify different facial expressions.

This paper presents a contactless methodology for detecting facial signs related to stress. A brief literature review shows that there are specific facial signs that are related to stressful conditions. A methodology based on computer vision techniques applied to color videos in order to extract facial signs such as movement (head), eyebrow lowering and raising, and blink rate is presented. Various facial features are investigated and are quantitatively evaluated through a frame-to-frame analysis. Preliminary results with few volunteers reveal a direct correlation of the selected facial signs with stress conditions. These initial results confirm our hypothesis regarding the possibility of assessing stress from facial expressions. However, these preliminary results should be quantitatively verified on a more comprehensive dataset containing a significant number of subjects in order to validate the algorithm results.

One of the significant aspects of the communication is its nonverbal aspect. In this study there was an attempt to explore this aspect of language teaching qualitatively from the learners’ point of view through observation and interview in Iranian context. It was revealed that four main categories of nonverbal communication(NVC) including: hand gesture, eye-contact, facial expression and tone of voice are used from the most frequently used to the least used respectively. Moreover, it was found that a teacher’s use of nonverbal communication can help learners to get more benefit from the learning context. Also, the findings indicate how utilizing NVC may contribute to different aspects (vocabulary, grammar, and pronuciation) of language learning. According to the students’ comments, six main themes as different roles of the teacher were found including: Teacher as a composer, Teacher as a constructor, Teacher as a mirror, Teacher as a conductor, Teacher as a mentor, and Teacher as a vocalist. Keywords: non-verbal, communication, lanugage learning, qualitative study, perception.

Coronavirus Disease 2019 (COVID-19), Videoconferencing, and Gender

Intelligent human-computer interaction (HCI) integrates versatile tools such as perceptual recognition, machine learning, affective computing, and emotion cognition to enhance the ways humans interact with computers. Facial expression analysis is one of the essential medium of behavior interpretation and emotion modeling. In this paper, we modify and develop a reconstruction method utilizing Principal Component Analysis (PCA) to perform facial expression recognition. A framework of hierarchical radial basis function network (HRBFN) is further proposed to classify facial expressions based on local features extraction by PCA technique from lips and eyes images. It decomposes the acquired data into a small set of characteristic features. The objective of this research is to develop a more efficient approach to discriminate between seven prototypic facial expressions, such as neutral, smile, anger, surprise, fear, disgust, and sadness. A constructive procedure is detailed and the system performance is evaluated on a public database ”Japanese Females Facial Expression (JAFFE).” We conclude that local images of lips and eyes can be treated as cues for facial expression. As anticipated, the experimental results demonstrate the potential capabilities of the proposed approach.

Interactive agents are becoming increasingly common in many application domains, such as education, healthcare and personal assistance. The success of such embodied agents relies on their ability to have sustained engagement with their human users. Such engagement requires agents to be socially intelligent, equipped with the ability to understand and reciprocate both verbal and non-verbal cues. While there has been tremendous progress in verbal communication, mostly driven by the success of speech recognition and question-answering, teaching agents to appropriately react to facial expressions has received less attention. In this paper, we focus on non-verbal facial cues for face-to-face communication between a user and an embodied agent. We propose a method that automatically learns to update the agent's facial expressions based on the user's expressions. We adopt a learning scheme and train a deep neural network on hundreds of videos, containing pairs of people engaging in a conversation, and without external human supervision. Our experimental results show the efficacy of our model in sustained long-term prediction of the agent's facial landmarks. We present comparative results showing that our model significantly outperforms baseline approaches and provide insightful human studies to better understand our model's qualitative performance. We release our dataset to further encourage research in this field.

This thesis considers the act of music performance quality evaluation as a performance in itself, examining the processes as well as the products of evaluative decision-making. It provides new understanding of performance evaluation through two experimental studies, two field surveys, and the development of a new mode to study and train evaluative skills. In the first study (Chapter 3), 42 musicians provided continuous quality evaluations of five piano works by Chopin and a twentieth-century composer varying by length and familiarity. Three of these pieces had been manipulated to contain performance errors in the opening material, and two of those the same error at the recapitulation. Results showed that familiarity had no effect within works of a well-known composer, but times to first and final decision were significantly extended for an unfamiliar work of an unfamiliar composer. A shorter piece led to a shorter time to first decision. An error at the beginning of a performance caused a shorter time to first decision and lower initial and final ratings, where the same error at the recapitulation did not have a significant effect on the final judgement, despite causing a temporary negative drop. In the second study (Chapter 4), 53 musicians and 52 non-musicians gave continuous quality evaluations of one of five randomly assigned videos manipulated to include an inappropriate stage entrance, aural performance error, error with negative facial reaction, or facial reaction alone. Results showed that participants viewing the ‘inappropriate’ stage entrance made judgements significantly more quickly than those viewing the ‘appropriate’ entrance. The aural error caused an immediate drop in quality judgements that persisted to a lower final score only when accompanied by the frustrated facial expression from the pianist; the performance error alone caused a temporary drop only in the musicians’ ratings, and the negative facial reaction alone caused no reaction regardless of participants’ musical experience. The two survey studies comprised custom questionnaires delivered to large audiences (300 & 433) in live professional settings. The first survey (Chapter 5) examined the relationship between self-reported mood and anxiety states before and after performance with perceived quality and enjoyment of the music. The second (Chapter 6) expanded this to incorporate individuals’ perceptions of the social and physical environment. Results from both studies found high correlations of enjoyment and quality ratings, with familiarity with the music not predictive of either outcome. Mood states following the performance were more predictive of judgements than those reported prior. Seat location was not predictive of perceptions, although ratings of the building’s acoustic and appropriateness were moderately predictive. Concertgoers assumed their own ratings to be marginally higher than those of their fellow audience members. Based on the challenges faced in studying performance evaluation in ecologically valid settings, and the parallel difficulties in training the skill of performing evaluations, the principles of Immersive Virtual Environments and distributed simulation are discussed as potential solutions through the proposal of the Evaluation Simulator (Chapter 7). All results of the thesis are then discussed concerning their implications for musicians, teachers, and organisations, as well as domains beyond music, in executing and training effective evaluations of human performance. A new research agenda is posited that examines the act of performance evaluation with the same rigour and consideration of complexity given to the performances themselves.

Three-dimensional human face modeling is a very popular topic with many applications,such as facial animation, face recognition and model-based facial video communication. Therefore, how to model the facial geometry, texture intensity and illumination variation is important in computer vision and graphics. Previous works on 3D head modeling from a single face image utilized prior information on 3D head models. However, it is difficult to accurately reconstruct the 3D face model from a single face image with expression since the facial expression induces 3D face model deformation in a complex manner. The main challenge is the coupling of the neutral 3D face model and the 3D deformation due to expression, thus making the 3D model estimation from a single face image with expression very challenging. On the other hand, the illumination condition also makes the problem more difficult. In this thesis,we focus on developing a 3D face model reconstruction system including surface registration and training of 3D face models with expressional deformations as well as the estimation of the 3D neutral shape and the 3D expressional deformation from a single face image. The proposed reconstruction algorithm integrates the linear and non-linear subspace representations for a prior 3D neutral morphable model and the probabilistic manifold-based 3D expressional deformation. We incorporate the face geometry, expression deformation, texture and illumination information into the problem so that it is well constrained. The reconstructed 3D face models can also be further extended and applied to many real-world applications.

Nonverbal communication is a crucial element of interpersonal interactions within professional settings, playing a significant role in shaping perceptions, relationships, and overall communication effectiveness. This article examines various dimensions of nonverbal communication, including body language, facial expressions, gestures, posture, eye contact, and proxemics. It explores how these elements influence professional relationships, negotiations, leadership dynamics, and team collaboration. Through a review of existing literature, practical examples, and case studies, the article highlights the interplay between nonverbal cues and verbal messages, illustrating how nonverbal communication can reinforce or contradict spoken words. The findings underscore the importance of developing nonverbal awareness and skills among professionals to foster effective interactions, enhance trust and credibility, and promote successful outcomes in diverse workplace environments. Ultimately, the article advocates for a more comprehensive understanding of nonverbal communication as a key component in achieving professional success.

The goal of this review was to identify and summarize technology-assisted methods that assess nonverbal behaviors known to influence the quality of the therapeutic alliance between health care providers and patients in clinical, research, or educational settings. Strong therapeutic alliances in health care provider-patient relationships contribute to positive patient outcomes. Previous studies have shown that nonverbal behaviors from providers, such as tone of voice, facial expression, eye contact, and physical positioning, help build strong therapeutic relationships. Technological advances have created opportunities to automate detection and feedback about nonverbal behaviors in clinical interactions without the intervention of highly trained instructors. Automated detection and feedback will increase opportunities to develop better nonverbal communication, which is expected to improve the quality of the therapeutic alliance. Participants were current or aspiring health care providers who routinely would be expected to have patient encounters in clinical, educational, or research settings. Interactions could include actual patient interactions, model patients, virtual patients, or simulated patients. Included studies involved technology-assisted methods to collect or analyze at least 1 nonverbal behavior. This review included qualitative and quantitative studies and review articles. PubMed, Embase, CINAHL, ERIC, Scopus, and Google Scholar were searched for published and unpublished literature in English between 2010 and 2023. Two reviewers independently completed title and abstract screening, full-text review, and data extraction. Any conflicts that arose were resolved by a third reviewer. Extraction was completed by 2 reviewers. Twenty-five sources were included. The most frequently measured behaviors included gaze frequency or duration on the patient's face (52% of studies) and facial expression (48%), with 24% of studies measuring social touch or speech patterns, 20% measuring posture and gestures, 12% measuring proximity to patient, 8% measuring synchrony, and 4% measuring voice. Technologies for data collection included audio or video recordings and other technologies such as eye-tracking glasses or multi-functional headsets, used alone or integrated into augmented reality or virtual reality systems. Challenges associated with analyzing data included recognition of patients' and providers' voices, facial features, and body parts, and classifying body posture and movement as well as facial expression and movements. Studies used commercial, open source, or customized software to meet each challenge. Many incorporated machine learning or algorithms associated with artificial intelligence (AI), which required significant training data. Technology-assisted methods for analyzing voice and social touch were limited, whereas there were several available analysis techniques for analyzing synchrony, gestures, posture, and proximity to patients. The use of technology in collection and analysis of provider-patient interactions to enhance nonverbal behaviors and communication skills is rapidly developing, with multiple options available. Future efforts should focus on technology-assisted methods to analyze voice and social touch, and should transparently report contributing factors and training data for machine learning models or AI-associated algorithms. Development of systems will require significant testing to ensure accessibility to the intended audience and broadly accurate and meaningful nonverbal behaviors. OSF ( https://osf.io/kedmf ).

Nonverbal expressiveness and EMG activity in tension headache sufferers and controls

Nonverbal communication, including gestures, facial expressions, eye contact, and body movements, serves as a valuable tool in English as a Foreign Language classrooms by reinforcing language comprehension and fostering an engaging learning environment. This study explores the role of nonverbal communication in enhancing reading comprehension among Vietnamese students learning English as a Foreign Language. A mixed-methods approach is employed, incorporating both quantitative and qualitative data gathered through observations and student questionnaires in the English as a Foreign Language classroom setting at a Vietnamese university. The findings indicate that gestures and facial expressions are particularly effective in aiding vocabulary comprehension and interpreting the emotional tone of reading material. Eye contact and body movements contributed significantly to maintaining focus and creating a supportive atmosphere that encourages student participation. These insights suggest that integrating nonverbal communication strategies into reading instruction can make language more accessible and relatable, enhancing students’ comprehension and engagement. This study underscores the importance of nonverbal cues in English as a Foreign Language pedagogy and offers recommendations for further research on nonverbal communication’s role across different language learning skills.

The paper proposes a method to acquire nonverbal cues for an intelligent agent that can be used in communications with human partner. A framework to extract the information that affords a key to understand intension, mental state, affect, or signs evoked by a person through nonverbal communication between the person and an intelligent agent. The aim of the framework is to make intelligent agents to be able to extract the nonverbal cues through trial-and-error style learning. A limited functionality of the framework is implemented and applied to a facial expression analysis as a case study. The main advantage of the method is that it would acquire the cues as some time-series information inherent to particular facial expressions. Thus the agent Result in a preliminary experiment shows that this implemented method have potential to extract nonverbal cues concerning with fake and real smiles.

This paper introduces a framework, called <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EMOTION</monospace>, for generating expressive motion sequences in humanoid robots, enhancing their ability to engage in human-like non-verbal communication. Non-verbal cues such as facial expressions, gestures, and body movements play a crucial role in effective interpersonal interactions. Despite the advancements in robotic behaviors, existing methods often fall short in mimicking the diversity and subtlety of human non-verbal communication. To address this gap, our approach leverages the in-context learning capability of large language models (LLMs) to dynamically generate socially appropriate gesture motion sequences for human-robot interaction. We use this framework to generate 10 different expressive gestures and conduct online user studies comparing the naturalness and understandability of the motions generated by <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EMOTION</monospace> and its human-feedback version, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EMOTION++</monospace>, against those by human operators. The results demonstrate that our approach either matches or surpasses human performance in generating understandable and natural robot motions under certain scenarios. We also provide design implications for future research to consider a set of variables when generating expressive robotic gestures.