Realistic Facial Expression Reconstruction for VR HMD Users

Virtual reality (VR) games are currently becoming part of the public-space entertainment (e.g., VR amusement parks). Therefore, VR games should be attractive for players, as well as for bystanders. Current VR systems are still mostly focused on enhancing the experience of the head-mounted display (HMD) users; thus, bystanders without an HMD cannot enjoy the experience together with the HMD users. We propose the "ReverseCAVE": a proof-of-concept prototype for public VR visualization using CAVE-based projection with translucent screens for bystanders toward a shareable VR experience. The screens surround the HMD user and the VR environment is projected onto the screens. This enables the bystanders to see the HMD user and the VR environment simultaneously. We designed and implemented the ReverseCAVE, and evaluated it in terms of the degree of attention, attractiveness, enjoyment, and shareability, assuming that it is used in a public space. Thus, we can make the VR world more accessible and enhance the public VR experience of the bystanders via the ReverseCAVE.

While the eyes are very important in human communication, once a user puts on a head mounted display (HMD), the face is obscured from the outside world's perspective. This leads to communication problems when bystanders approach or collaborate with an HMD user. We introduce transparentHMD, which employs a head-coupled perspective technique to produce an illusion of a transparent HMD to bystanders. We created a self contained system, based on a mobile device mounted on the HMD with the screen facing bystanders. By tracking the relative position of the bystander using the smartphone's camera, we render an adapting perspective view in realtime that creates the illusion of a transparent HMD. By revealing the user's face to bystanders, our easy to implement system allows for opportunities to investigate a plethora of research questions particularly related to collaborative VR systems.

Facial action unit (AU) recognition is an important task for facial expression analysis. Traditional AU recognition methods typically include a supervised training, where the AU annotated training images are needed. AU annotation is a time consuming, expensive, and error prone process. While AU is hard to annotate, facial expression is relatively easy to label. To take advantage of this, we introduce a new learning method that trains an AU classifier using images with incomplete AU annotation but with complete expression labels. The goal is to use expression labels as hidden knowledge to complement the missing AU labels. Towards this goal, we propose to construct a Bayesian Network (BN) to capture the relationships between facial expression and AUs. Structural Expectation Maximum is used to learn the structure and parameters of the BN when the AU labels are missing. Given the learned BNs and measurements of AUs and expression, we can then perform AU recognition within the BN through a probabilistic inference. Experimental results on the CK+ and ISL databases demonstrate the effectiveness of our method.

Emotion management is a critical psychological skill for athletes, deserving significant attention during the foundational training phase. Early intervention in athletes’ emotional states allows coaches to devise effective training plans, ultimately enhancing competition performance. Emotions can be evaluated through facial expressions and physiological signals, providing a basis for detecting athletes’ training emotions. This study explores the biomechanical characteristics of facial expressions to recognize emotions. First, the relationship between facial expressions and facial Action Units (AUs) is analyzed, identifying AU combinations that effectively represent facial expressions. Considering the temporal and spatial dynamics of facial expressions, AUs are used as feature inputs for an SVM model to classify athletes’ training emotions. Subsequently, emotional expressions are mapped to corresponding emotional states, introducing an emotion index to quantify athletes’ training emotions. Finally, quantitative emotion recognition is achieved based on transient facial expressions observed during training. This research provides a theoretical framework for coaches and athletes to enhance emotional management through biomechanically informed training approaches.

Human psychological stress is a vast and highly complicated topic of study and research. The types and kinds of stress observed in humans vary among researchers. Also, to identify stress, many methods exist. Most of these methods are non-intrusive and are based on self-reporting and questionnaires which reduces the real-time efficacy of the procedure. Intrusive methods are, on the other hand, time consuming and cumbersome. The total problem of non-intrusive psychological stress detection from facial images can be visualized in three incremental stages: instantaneous analysis of subject, historical analysis of subject, and the subject’s environmental analysis. In this paper, we deal with instantaneous analysis of a subject. This means that the stress behavior of a subject is predicted for one moment of time using an image of his/her facial expression. In order to do so, we have conducted two surveys to establish the relationship between emotional compositions of a facial expression with stress and also to establish the relationship of individual emotions with stress. The novelty of the paper is 1) to establish relationships between the seven basic emotions (anger, contempt, disgust, fear, happy, sad, and surprise) and stress, 2) to establish relationship between emotional composition of a facial expression and stress, and 3) to predict a formula for evaluating stress in terms of emotional percentage mixture of a facial expression. In order to achieve the three goals, we use Facial Action Unit (AU) [1] coded image data to predict the emotional mixture of the facial expression in terms of the seven basic emotion percentages. An AU represents one of the many basic muscle movements that make up the facial expression. Then we analyze the survey outcomes to establish the relationship between individual emotions and stress. Finally we correlate the survey outcomes with the emotional mixture data obtained from the facial expression using Hidden Markov Model (HMM) approach to both establish a relationship of emotional composition with stress and to predict a formula for stress in terms of the seven basic emotion percentages jointly.

The purpose of this research was to identify the existence of facial and body expressions that reflects the emotional state of dairy calves. Thirty-two calves between 1 and 8 weeks of age were studied for four consecutive days. Eight groups with four animals each were formed randomly, forming homogeneous groups among themselves in age. Each group was analyzed twice. The animals were taken to a test chamber with open sides and covered with metal tile over a pasture area, where they received the application of the treatments positive "neck stroking" and negative "umbrella test" (Sandem et al. 2004). The facial expression was analyzed through the facial action units: eyebrow, mouth, nostril and eye white. Body expression was verified by the ear position and tail position variables. The animals were evaluated during six experimental times: natural environment, pre-test, test, post-test 1, post-test 2 and natural environment after the test. Both stimuli caused reduction (p < 0,05) in eye white area from 3,50 cm 2 to 1,10 cm 2 , and area of nostril from 8,58 cm 2 to 6,07 cm 2 . The raised eyebrow was present during the negative stimulus. It was not possible to identify if the changes in facial action units were due to positive or negative emotions. All manipulation with the animals caused a change in the facial units. The positioning of ear and tail are related to positive and negative emotions. The "back" ear position and the "arched" tail position are associated with expressing positive emotions. The "horizontal" ear position and the "between hindlimbs" tail position, to negative emotions. The types of ear and tail positions are indicators of positive or negative emotional state in calves, and there is great potential for using these measures to evaluate emotions. Thus, we identified the existence of facial expressions, with the variable eyebrow, and body expression, with the position of "back" ear and "arched" tail that reflects positive emotional states; and "horizontal" ear position and "between hindlimbs" tail reflecting negative emotional states in calves.

Noise is annoying. Facial expressions indicate diverse emotions instantly, objectively and accurately. In this paper, the facial emotional expression is used to assess the noise annoyance to nullify the disadvantages of traditional evaluation methods. Firstly, by exploiting listening experiments, the facial videos, physiological signals and individual annoyance from 30 listeners with different nationalities were collected. Then, the facial action units (AUs) were extracted, identified and analyzed, and the AUs combination reflects annoyed emotion induced by the noise was confirmed. Furthermore, a novel evaluation scale embodying color and emotional symbols was proposed, which has been proven to be valid by the social surveys. The present study confirms that it is feasible to evaluate the noise annoyance more objectively by exploring the listener's facial "annoyance" expression. The innovative scale can avoid the understanding bias and improve the evaluation efficiency remarkably.

In this demonstration, we introduce Face/On, an embedded feedback device that leverages the contact area between the user's face and a virtual reality (VR) head-mounted display (HMD) to provide rich haptic feedback in virtual environments (VEs). Head-worn haptic feedback devices have been explored in previous work to provide directional cues via grids of actuators and localized feedback on the users' skin. Most of these solutions were immersion breaking due to their encumbering and uncomfortable design and build around a single actuator type, thus limiting the overall fidelity and flexibility of the haptic feedback. We present Face/On, a VR HMD face cushion with three types of discreetly embedded actuators that provide rich haptic feedback without encumbering users with invasive instrumentation on the body. By combining vibro-tactile and thermal feedback with electrical muscle stimulation (EMS), Face/On can simulate a wide range of scenarios and benefit from synergy effects between these feedback types.

Automatic facial expression analysis is an important aspect of human machine interaction as the face is an important communicative medium. We use our face to signal interest, disagreement, intentions or mood through subtle facial motions and expressions. Work on automatic facial expression analysis can roughly be divided into the recognition of prototypic facial expressions such as the six basic emotional states and the recognition of atomic facial muscle actions (action units, AUs). Detection of AUs rather than emotions makes facial expression detection independent of culture-dependent interpretation, reduces the dimensionality of the problem and reduces the amount of training data required. Classic psychological studies suggest that humans consciously map AUs onto the basic emotion categories using a finite number of rules. On the other hand, recent studies suggest that humans recognize emotions unconsciously with a process that is perhaps best modeled by artificial neural networks (ANNs). This paper investigates these two claims. A comparison is made between detection of emotions directly from features vs. a two-step approach where we first detect AUs and use the AUs as input to either a rulebase or an ANN to recognize emotions. The results suggest that the two-step approach is possible with a small loss of accuracy and that biologically inspired classification techniques outperform those that approach the classification problem from a logical perspective, suggesting that biologically inspired classifiers are more suitable for computer-based analysis of facial behavior than logic inspired methods

The inherent dependencies among facial action units (AU) caused by the underlying anatomic mechanism are essential for the proper recognition of AUs and estimation of intensity levels, but they have not been exploited to their full potential. We are proposing novel methods to recognize AUs and estimate intensity via hybrid Bayesian networks. The upper two layers are latent regression Bayesian networks (LRBNs), and the lower layers are Bayesian networks (BNs). The visible nodes of the LRBN layers are representations of ground-truth AU occurrences or AU intensities. Through the directed connections from latent layer and visible layer, an LRBN can successfully represent relationships between multiple AUs or AU intensities. The lower layers include Bayesian networks with two nodes for AU recognition, and Bayesian networks with three nodes for AU intensity estimation. The bottom layers incorporate measurements from facial images with AU dependencies for intensity estimation and AU recognition. Efficient learning algorithms of the hybrid Bayesian networks are proposed for AU recognition as well as intensity estimation. Furthermore, the proposed hybrid Bayesian network models are extended for facial expression-assisted AU recognition and intensity estimation, as AU relationships are closely related to facial expressions. We test our methods on three benchmark databases for AU recognition and two benchmark databases for intensity estimation. The results demonstrate that the proposed approaches faithfully model the complex and global inherent AU dependencies, and the expression labels available only during training can boost the estimation of AU dependencies for both AU recognition and intensity estimation.

IntroductionCardiovascular Emotional Dampening (CED) refers to blunted emotional responsiveness in individuals with elevated blood pressure (BP), but research has exclusively focused on how such individuals perceive others’ emotions. Given evidence that the ability to produce facial expressions of emotions is closely tied to emotion recognition via shared neural mechanisms, examining expressive deficits in CED could reveal additional pathways linking elevated BP with emotional communication. This study examined whether individuals with higher systolic and diastolic BP exhibit reduced accuracy and intensity when generating prototypical facial expressions of emotion.MethodsParticipants (N = 74) across normotensive (n=33), prehypertensive (n=21), and hypertensive (n=20) categories were instructed to pose six basic emotions. Facial Action Units (AUs) were coded using certified human coders and OpenFace, allowing comparison of AU intensities, human-machine agreement, and expression accuracy. Prototypical emotion templates were used to determine accuracy, and interrater agreement was quantified via five complementary indices.ResultsEmotional expression accuracy was significantly lower in prehypertensive and hypertensive groups, particularly for sadness, fear, and surprise. Correlational analyses revealed significant negative associations of SBP and DBP with accuracy of expressing sadness, disgust, and anger. Notably, expressions of happiness were preserved. Although overall agreement between human and machine ratings was high, reduced intensity and increased AU sparsity at higher BP levels likely suppressed reliability metrics.DiscussionThese findings extend the CED framework from recognition to expression, revealing that elevated BP may blunt the expressive channel for particularly negative emotions, irrespective of arousal level. Though facial expression of emotions is limited to negative emotions, it is generalized across different levels of arousal within this category - from low (sad), moderate (disgust), to high (anger).The pattern suggests central autonomic influences on facial expressivity of emotions and opens new directions for identifying emotional communication deficits in at risk populations.

In this cross-sectional observational study, we developed a computer vision-based machine-learning model to estimate headache pain intensity from facial action units (AUs) and evaluated its association with self-reported pain. Facial videos were obtained from 80 adults aged 19-80 years receiving headache care at a single institution (outpatient, inpatient, or emergency; IRB No. 2024-08-003). A multitask cascaded convolutional network was used to detect and align faces, and convolutional neural networks estimated facial landmark locations and AU intensities. Recordings were synchronized and denoised. We extracted an APEX frame representing the peak facial expression and computed a headache pain intensity index (HPII) by combining pain-relevant AUs. The HPII showed a moderate positive Pearson correlation coefficient (r) with visual analog scale scores (r = 0.413-0.522, where higher r values indicate a stronger linear association) among participants with moderate-to-severe pain. This AU-based apex-frame approach may serve as a practical nonverbal indicator for monitoring headache pain intensity.

The prefrontal cortex plays a key role in emotional state. Electroencephalography (EEG) studies have reported relationships between frontal asymmetry in the alpha band, emotional state, and emotion-related motivation. The current study investigated whether the positive or negative valence of emotional stimulation or the behavioral intention to either facilitate or suppress one’s facial expression in response to these stimuli is reflected in relevant changes in frontal EEG asymmetry. EEG was recorded while participants either produced a facial expression that was in accord with positive or negative feelings corresponding to image stimuli, or suppressed their facial expressions. The laterality index of frontal alpha power indicated greater relative right frontal activity while participants suppressed facial expression compared with facilitating facial expression during emotional stimulation. However, there was no difference in frontal asymmetry between the presentation of image stimuli showing facial expressions corresponding to positive vs. negative emotions. These results suggested that frontal asymmetry was related to the control of facial emotional expressions rather than the perception of positive vs. negative emotions. Moreover, microstate analysis revealed that the appearance rate of microstate class B with polarity in the left frontal area increased during the suppression of facial expressions. The present results suggested that frontal asymmetry reflects the control of facial emotional expressions, which supports the motivational direction model.

An efficient, global and local image-processing based extraction and tracking of intransient facial features and automatic recognition of facial expressions from both static and dynamic 2D image/video sequences is presented. Expression classification is based on Facial Action Coding System (FACS) a lower and upper face action units (AUs), and discrimination is performed using Probabilistic Neural Networks (PNN) and a Rule-Based system. For the upper face detection and tracking, we use systems based on a novel two-step active contour tracking system while for the upper face, cross-correlation based tracking system is used to detect and track of Facial Feature Points (FFPs). Extracted FFPs are used to extract some geometric features to form a feature vector which is used to classify input image or image sequences into AUs and basic emotions. Experimental results show robust detection and tracking and reasonable classification where an average recognition rate is 96.11% for six basic emotions in facial image sequences and 94% for five basic emotions in static face images.KeywordsActive contoursAction UnitsFacial ExpressionsProbabilistic Neural Networks

Recently, an effective method to solve the problem of complex facial expression recognition is to encode individual facial expressions through the action units (AUs) encoded by the Facial Action Coding System (FACS). A large number of methods have been proposed for AU detection, but the problem of AU detection is still very challenging due to the different sizes and shapes of facial AU. Therefore, to solve the problem of locating AU, many methods first detect the landmark of the face and then detect AU according to the location of the landmark. However, in this way, the accuracy of landmark detection greatly affects the detection result of AU, making this method less robust. In this paper, we use a multi-scale structure (MTL) to solve the problem of facial AU region distribution size varies while being independent of the facial landmark information. Besides, due to the strong correlation between facial AU regions, we propose the structure of region relationship learning (LRT) using rich local information to learn the relationship between facial local regions. The end-to-end Multi-Scale Region with Local Relationship Learning (MTLRT-Net) we proposed is a lite network with low hardware requirements, and extensive experiments on the BP4D and DISFA demonstrate that our network framework is better than the state-of-the-art methods.