An optimized ERP brain–computer interface based on facial expression changes

Interferences from the spatially adjacent non-target stimuli evoke ERPs during non-target sub-trials and lead to false positives. This phenomenon is commonly seen in visual attention based BCIs and affects the performance of BCI system. Although, users or subjects tried to focus on the target stimulus, they still could not help being affected by conspicuous changes of the stimuli (flashes or presenting images) which were adjacent to the target stimulus. In view of this case, the aim of this study is to reduce the adjacent interference using new stimulus presentation pattern based on facial expression changes. Positive facial expressions can be changed to negative facial expressions by minor changes to the original facial image. Although the changes are minor, the contrast will be big enough to evoke strong ERPs. In this paper, two different conditions (Pattern_1, Pattern_2) were used to compare across objective measures such as classification accuracy and information transfer rate as well as subjective measures. Pattern_1 was a “flash-only” pattern and Pattern_2 was a facial expression change of a dummy face. In the facial expression change patterns, the background is a positive facial expression and the stimulus is a negative facial expression. The results showed that the interferences from adjacent stimuli could be reduced significantly (P<0.05) by using the facial expression change patterns. The online performance of the BCI system using the facial expression change patterns was significantly better than that using the “flash-only” patterns in terms of classification accuracy (p<0.01), bit rate (p<0.01), and practical bit rate (p<0.01). Subjects reported that the annoyance and fatigue could be significantly decreased (p<0.05) using the new stimulus presentation pattern presented in this paper.

The ability of 7- and 10-month-old infants to generalize their discrimination of facial expressions lacking featural consistency was investigated in a series of 4 experiments. Infants were habituated to models posing either prototypically positive displays (e.g., happy expressions) or positive expression blends (e.g., mock surprise). They were then tested for their ability to recognize a positive expression on the face of a novel model and to discriminate positive and negative facial expressions. Only in Experiment 1, in which infants were familiarized to a mix of happy and surprised facial expressions, did 10-month-olds demonstrate generalized discrimination of positive affect. When positive blends or happy expressions alone served as familiarization stimuli, both 7- and 10-month-old infants failed to dishabituate to a change in affective tone. 7-month-olds, in particular, showed consistent recovery of looking to the introduction of novel models. The pattern of results suggests that it is not until sometime after 7 months of age that dependence on the presence of expression-specific features for affect recognition and discrimination diminishes. By 10 months of age, however, infants are beginning to recognize the affective similarity of familiar positive facial expressions.

There are two major parallel pathways in humans: the parvocellular (P) and magnocellular (M) pathways. The former has excellent spatial resolution with color selectivity while the latter shows excellent temporal resolution with high contrast sensitivity. Visual stimuli should be tailored to answer specific clinical and/or research questions. We examined the neural mechanisms of face perception using event-related potentials (ERPs). Face stimuli of different spatial frequencies were used to investigate how low-spatial-frequency (LSF) and high-spatial-frequency (HSF) components of the face contribute to the identification and recognition of the face and facial expressions. The P100 in the occipital area (Oz), N170 in the posterior temporal region (T5/T6) and late components peaking at 270–390 ms (T5/T6) were analyzed. LSF enhanced P100 while N170 was augmented by HSF irrespective of facial expressions, which suggests that LSF is important for global processing for face whereas HSF handles featural processing. There were significant amplitude differences between positive and negative LSF facial expressions in the early time windows of 270–310 ms. Subsequently, the amplitudes among negative HSF facial expressions differed significantly in the later time windows of 330–390 ms. Discrimination between positive and negative facial expressions precedes discrimination among different negative expressions in a sequential manner based on parallel visual channels. Interestingly, patients with schizophrenia showed decreased spatial frequency sensitivities for face processing. Taken together, our spatially filtered face images are useful for exploring the face perception and recognition.

Over the course of the last decade, infrared (IR) and particularly thermal IR imaging based face recognition has emerged as a promising complement to conventional, visible spectrum based approaches which continue to struggle when applied in the real world. While inherently insensitive to visible spectrum illumination changes, IR images introduce specific challenges of their own, most notably sensitivity to factors which affect facial heat emission patterns, e.g. emotional state, ambient temperature, and alcohol intake. In addition, facial expression and pose changes are more difficult to correct in IR images because they are less rich in high frequency detail which is an important cue for fitting any deformable model. In this paper we describe a novel method which addresses these major challenges. Specifically, to normalize for pose and facial expression changes we generate a synthetic frontal image of a face in a canonical, neutral facial expression from an image of the face in an arbitrary pose and facial expression. This is achieved by piecewise affine warping which follows active appearance model (AAM) fitting. This is the first publication which explores the use of an AAM on thermal IR images; we propose a pre-processing step which enhances detail in thermal images, making AAM convergence faster and more accurate. To overcome the problem of thermal IR image sensitivity to the exact pattern of facial temperature emissions we describe a representation based on reliable anatomical features. In contrast to previous approaches, our representation is not binary; rather, our method accounts for the reliability of the extracted features. This makes the proposed representation much more robust both to pose and scale changes. The effectiveness of the proposed approach is demonstrated on the largest public database of thermal IR images of faces on which it achieved 100% identification rate, significantly outperforming previously described methods.

This study, for the first time investigates facial expressions during the retrieval of self-defining memories (i.e., vivid and emotionally intense memories of enduring concerns or unresolved conflicts). Participants self-rated the emotional valence of their self-defining memories, while autobiographical retrieval was analyzed with facial analysis software. This software (Facereader) synthesizes the information in facial expression (i.e., cheek, lips, muscles, eyebrow muscles) to describe and categorize facial expressions (i.e., neutral, happy, sad, surprised, angry, scared, and disgusted facial expressions). It was found that participants showed more emotional than neutral facial expressions during the retrieval of self-defining memories. It was also found that participants showed more positive than negative facial expressions during the retrieval of self-defining memories. Interestingly, participants attributed positive valence to the retrieved memories. These findings are the first to demonstrate the consistency between facial expressions and the subjective emotional experience of self-defining memories. These findings provide valuable physiological information about the emotional experience of the past.

Facial expression is an integral aspect of non-verbal communication of affective information. Earlier psychological studies have reported that the presentation of prerecorded photographs or videos of emotional facial expressions automatically elicits divergent responses, such as emotions and facial mimicry. However, such highly controlled experimental procedures may lack the vividness of real-life social interactions. This study incorporated a live image relay system that delivered models’ real-time performance of positive (smiling) and negative (frowning) dynamic facial expressions or their prerecorded videos to participants. We measured subjective ratings of valence and arousal and facial electromyography (EMG) activity in the zygomaticus major and corrugator supercilii muscles. Subjective ratings showed that the live facial expressions were rated to elicit higher valence and more arousing than the corresponding videos for positive emotion conditions. Facial EMG data showed that compared with the video, live facial expressions more effectively elicited facial muscular activity congruent with the models’ positive facial expressions. The findings indicate that emotional facial expressions in live social interactions are more evocative of emotional reactions and facial mimicry than earlier experimental data have suggested.

Reinforcement values of facial expressions for visual fixation remain underdeveloped in extant literature, though numerous studies have investigated the function of facial expressions as antecedents of gaze behavior. This study used a concurrent gaze-contingent reinforcement procedure to compare reinforcement values of positive and negative facial expressions for gaze fixation. In total, 7 of the 20 typical adults fixated more frequently on the side that was followed by positive facial expressions. The results suggest that a novel application of eye tracking technology could allow for the evaluation and comparison of reinforcement values of social visual stimuli.

A bi-modal face recognition framework integrating facial expression with facial appearance

Facial expressions are among the earliest behaviors infants use to express emotional states, and are crucial to preverbal social interaction. Manual coding of infant facial expressions, however, is laborious and poses limitations to replicability. Recent developments in computer vision have advanced automated facial expression analyses in adults, providing reproducible results at lower time investment. Baby FaceReader 9 is commercially available software for automated measurement of infant facial expressions, but has received little validation. We compared Baby FaceReader 9 output to manual micro-coding of positive, negative, or neutral facial expressions in a longitudinal dataset of 58 infants at 4 and 8 months of age during naturalistic face-to-face interactions with the mother, father, and an unfamiliar adult. Baby FaceReader 9’s global emotional valence formula yielded reasonable classification accuracy (AUC = .81) for discriminating manually coded positive from negative/neutral facial expressions; however, the discrimination of negative from neutral facial expressions was not reliable (AUC = .58). Automatically detected a priori action unit (AU) configurations for distinguishing positive from negative facial expressions based on existing literature were also not reliable. A parsimonious approach using only automatically detected smiling (AU12) yielded good performance for discriminating positive from negative/neutral facial expressions (AUC = .86). Likewise, automatically detected brow lowering (AU3+AU4) reliably distinguished neutral from negative facial expressions (AUC = .79). These results provide initial support for the use of selected automatically detected individual facial actions to index positive and negative affect in young infants, but shed doubt on the accuracy of complex a priori formulas.

Cortical responses to dynamic emotional facial expressions generalize across stimuli, and are sensitive to task-relevance, in adults with and without Autism

BackgroundNeurological disorders, ranging from common conditions like Alzheimer’s disease that is a progressive neurodegenerative disorder and remains the most common cause of dementia worldwide to rare disorders such as Angelman syndrome, impose a significant global health burden. Altered facial expressions are a common symptom across these disorders, potentially serving as a diagnostic indicator. Deep learning algorithms, especially convolutional neural networks (CNNs), have shown promise in detecting these facial expression changes, aiding in diagnosing and monitoring neurological conditions.ObjectivesThis systematic review and meta-analysis aimed to evaluate the performance of deep learning algorithms in detecting facial expression changes for diagnosing neurological disorders.MethodsFollowing PRISMA2020 guidelines, we systematically searched PubMed, Scopus, and Web of Science for studies published up to August 2024. Data from 28 studies were extracted, and the quality was assessed using the JBI checklist. A meta-analysis was performed to calculate pooled accuracy estimates. Subgroup analyses were conducted based on neurological disorders, and heterogeneity was evaluated using the I2 statistic.ResultsThe meta-analysis included 24 studies from 2019 to 2024, with neurological conditions such as dementia, Bell’s palsy, ALS, and Parkinson’s disease assessed. The overall pooled accuracy was 89.25% (95% CI 88.75–89.73%). High accuracy was found for dementia (99%) and Bell’s palsy (93.7%), while conditions such as ALS and stroke had lower accuracy (73.2%).ConclusionsDeep learning models, particularly CNNs, show strong potential in detecting facial expression changes for neurological disorders. However, further work is needed to standardize data sets and improve model robustness for motor-related conditions.

Tuning functions for automatic detection of brief changes of facial expression in the human brain

Visual evoked potentials were recorded from the occipital, parietal, central, frontal, and posterior temporal areas of the cortex during recognition of emotionally positive, negative and neutral facial expressions and during passive observation in 22 right-handed healthy subjects. These studies showed that in the posterior temporal areas, the latencies of the N90, P150, and N180 waves of potentials evoked by faces with emotionally negative expressions were significantly shorter than those evoked by other types of facial stimuli. Differences were seen both on recognition and during passive observation. Correct recognition involved both hemispheres and was characterized by high levels of interhemisphere temporal correlation of the processes occurring during the development of the P150 wave in the posterior temporal and the N180 wave in the frontal parts of the cortex. The possible relationship of these data to primary subthreshold recognition of facial expressions in the posterior temporal fields of the cortex is discussed, as is the role of the frontal cortex in completing this process and in taking the correct decision about the nature of the image.

The ability to discriminate facial expressions of emotion is important for human communication and interaction. When this ability develops is largely unknown, with the origins believed to lie in infancy. Behavioural and brain-based evidence suggests that infants are capable of differentiating positive and negative facial expressions (i.e., sad vs. happy, surprised vs. angry), however there is little research examining whether infants can make more fine-grained discriminations among negative facial expressions (e.g., fearful vs. angry). In the present paper, two experiments use a novel technique known as Fast Periodic Visual Stimulation (FPVS) to assess discrimination of facial expressions by adults (n = 33) and 7-month-old infants (n = 33). Adults discriminated facial expressions, but 7-month-old infants did not. Reasons why infants did not show a discrimination response are explored and the potential benefits of FPVS are discussed.

Independent effects of working memory loads and facial expressions on event-related potential (ERP) responses: Evidence from mass univariate analysis.