Reading your own lips: Common-coding theory and visual speech perception

Introduction: To the best of our knowledge, there is a lack of reliable, validated, and standardized (Dutch) measuring instruments to document visual speech perception in a structured way. This study aimed to: (1) evaluate the effects of age, gender, and the used word list on visual speech perception examined by a first version of the Dutch Test for (Audio-)Visual Speech Perception on word level (TAUVIS-words) and (2) assess the internal reliability of the TAUVIS-words. Methods: Thirty-nine normal-hearing adults divided into the following 3 age categories were included: (1) younger adults, age 18–39 years; (2) middle-aged adults, age 40–59 years; and (3) older adults, age >60 years. The TAUVIS-words consist of 4 word lists, i.e., 2 monosyllabic word lists (MS 1 and MS 2) and 2 polysyllabic word lists (PS 1 and PS 2). A first exploration of the effects of age, gender, and test stimuli (i.e., the used word list) on visual speech perception was conducted using the TAUVIS-words. A mixed-design analysis of variance (ANOVA) was conducted to analyze the results statistically. Lastly, the internal reliability of the TAUVIS-words was assessed by calculating the Chronbach α. Results: The results revealed a significant effect of the used list. More specifically, the score for MS 1 was significantly better compared to that for PS 2, and the score for PS 1 was significantly better compared to that for PS 2. Furthermore, a significant main effect of gender was found. Women scored significantly better compared to men. The effect of age was not significant. The TAUVIS-word lists were found to have good internal reliability. Conclusion: This study was a first exploration of the effects of age, gender, and test stimuli on visual speech perception using the TAUVIS-words. Further research is necessary to optimize and validate the TAUVIS-words, making use of a larger study sample.

Virtual personal assistants (VPAs) like Siri and Alexa have become common objects in households. Users frequently rely on these systems to search the internet or help retrieve information. As such, it is important to know how using these products affect cognitive processes like memory. Previous research suggests that visual speech perception influences auditory perception in human-human interactions. However, many of these VPAs are designed as a box or sphere that does not interact with the user visually. This lack of visual speech perception when interacting with a VPA could affect the human interaction with a system and their retention of information such as determining how many ounces are in a cup or how to greet someone in another language. This poses the question of whether the design of these VPAs is preventing the ability of users to retain the information they get from these systems. To test this, we designed an experiment that will explore interactions between user memory and either a traditional audio presentation (as is found with Siri or Alexa, for example) or one that allows for visual speech perception. Participants were asked to listen to an audio clip of a nonsensical story. In one condition, participants were asked to listen while looking at a blank screen (analogous to the lack of visual feedback inherent when working with current VPA designs). After a block of 25 audio clips, the participants took a test on the information heard. This process was repeated with an animated face with synchronized mouth movements instead of a black screen. Other participants will experience the same two presentations, but in reverse order as to counterbalance condition presentation. Data collection is currently underway. We predicted that VPA paired with synchronized lip movement would promote visual speech perception and thus help participants retain information. While we are still collecting data, the trend currently does not show a significant difference between audio and lip movement conditions. This could be an indication of differing abilities in lipreading.

Background: Pathological and physiological disabilities of human auditory receptors reduce speech intelligibility under hearing loss (HL) at high frequencies. Objectives: The objective of the study is to investigate the speech-intelligibility-enhancement (SIE) in elderly with high-frequency HL (HFHL) through visual-speech-perception (VSP). The hypothesis is that the VSP of spondee words (two-syllable words with equal stress on both syllables) is properly recognized and reproduced with HFHL. Also; to decide the statistical relevance of the precise significant difference in speech discrimination (SD) with VSP on the subjected ear. Methods: Observational and descriptive studies of SIE with VSP on 12 elderly listeners (24 clear ears; without wax impaction) with progressive and bilateral HFHL were examined. The entire experimental records were checked for distribution with normal (Gaussian) using the Shapiro–Wilk's and paired Student's t-test (parametric test) had the 5% (0.05) significance level (α). Results: We analyzed, improved, and better performance in SIE with HFHL through VSP. The statistical P (probability) measure was calculated nothing, i.e., 0 (P < 0.05). Therefore, investigators strongly and carefully discarded the null hypothesis consideration. There was some significant value of statistical variation also exists with VSP. Moreover, with the help of the confidence-interval analysis, the same conclusion was achieved. Conclusions: In this study, it is concluded that the elderly-individuals of this experiment, whose mother tongue and official language is Hindi and English, respectively, they demonstrate the SIE with HFHL through VSP. This investigation also helps to improve the hearing sensitivity to some extent with VSP.

BackgroundLexical access refers to the generation or retrieval of conceptual representation. Representation of concepts or semantic knowledge involves the organization of concepts in an associative manner building up the mental lexicon. Several studies have pointed out an early presence of lexical access failure in Alzheimer’s Disease(AD). While lexical access deficit is rampant among patients with AD, the underlying processes involved in retrieval failure remain debatable. Some findings suggest that the difficulty in lexical access can be due to a deterioration in the conceptual representation. Therefore, to understand the nature of the difficulty underlying lexical access failure among people with AD, the current study aims to understand the involvement of semantic knowledge in lexical access.MethodIn this study, three groups of participants will be recruited, Healthy Controls (HC), Mild Cognitive Impairment (MCI) and Dementia with AD. According to apriori sample size calculation, a total sample size of 54 would be required for the present study, with 18 participants per group. In the current study, the semantic priming task is used to understand the ongoing debate about the involvement of semantic knowledge in Lexical access. The experiment consists of 4 tasks to probe their semantic knowledge and visual perception. Each trial consists of a prime‐ image or word, presented for some duration followed by a related or unrelated target. The participant needs to report whether the target was related to the prime or not.ResultThe lexical retrieval is expected to be differentiated along the different phases of the disease as a function of semantic priming. It is expected that the lexical retrieval in the HC group would be least impacted across the three conditions and AD with Dementia to be the most impacted. The performance on each condition could be differentiated across each disease severity group, along with a difference in the prime type, that is, primes related to the target would show better performance than unrelated primes across all the conditions in each of the groups.ConclusionThe findings from this study would contribute towards understanding processes underlying lexical access failure among people with Alzheimer’s Disease.

Although many audio-visual speech experiments have focused on situations where the presence of an incongruent visual speech signal influences the perceived utterance heard by an observer, there are also documented examples of a related effect in which the presence of an incongruent audio speech signal influences the perceived utterance seen by an observer. This study examined the effects that different distracting audio signals had on performance in a color and number keyword speechreading task. When the distracting sound was noise, time-reversed speech, or continuous speech, it had no effect on speechreading. However, when the distracting audio signal consisted of speech that started at the same time as the visual stimulus, speechreading performance was substantially degraded. This degradation did not depend on the semantic similarity between the target and masker speech, but it was substantially reduced when the onset of the audio speech was shifted relative to that of the visual stimulus. Overall, these results suggest that visual speech perception is impaired by the presence of a simultaneous mismatched audio speech signal, but that other types of audio distracters have little effect on speechreading performance.

A representation of abstract linguistic categories in the visual system underlies successful lipreading

From phonetic features to connected discourse, every level of psycholinguistic structure including prosody can be perceived through viewing the talking face. Yet a longstanding notion in the literature is that visual speech perceptual categories comprise groups of phonemes (referred to as visemes), such as /p, b, m/ and /f, v/, whose internal structure is not informative to the visual speech perceiver. This conclusion has not to our knowledge been evaluated using a psychophysical discrimination paradigm. We hypothesized that perceivers can discriminate the phonemes within typical viseme groups, and that discrimination measured with d-prime (d’) and response latency is related to visual stimulus dissimilarities between consonant segments. In Experiment 1, participants performed speeded discrimination for pairs of consonant-vowel spoken nonsense syllables that were predicted to be same, near, or far in their perceptual distances, and that were presented as natural or synthesized video. Near pairs were within-viseme consonants. Natural within-viseme stimulus pairs were discriminated significantly above chance (except for /k/-/h/). Sensitivity (d’) increased and response times decreased with distance. Discrimination and identification were superior with natural stimuli, which comprised more phonetic information. We suggest that the notion of the viseme as a unitary perceptual category is incorrect. Experiment 2 probed the perceptual basis for visual speech discrimination by inverting the stimuli. Overall reductions in d’ with inverted stimuli but a persistent pattern of larger d’ for far than for near stimulus pairs are interpreted as evidence that visual speech is represented by both its motion and configural attributes. The methods and results of this investigation open up avenues for understanding the neural and perceptual bases for visual and audiovisual speech perception and for development of practical applications such as visual lipreading/speechreading speech synthesis.

It has been proposed that the common metric between audio and visual speech perception could be just kinematics (say dynamics), a somewhat rather old gibsonian proposal illustrated until recently by moving point-lights displays. Results reported here address the same question in the frame of another paradigm than moving dots. Full images of the face were presented in moving and static conditions along a naturally silent vowel-to-vowel rounding gesture (our “silent pause lipreading paradigm”). In order to compare the benefit of motion vs. shape, the change in angle paradigm for face perception was used for feature/gesture identification. The question is then: Does viewer-centred optimisation of the projection of the rounding gesture override motion benefit? Such a saturation effect found for profiles supports a shape-from-shading processing for front views, enhanced by the shape-from-motion system, such systems being recruited in case of undersampling (point-lights) or for out of shape projections. These computational approaches avoid calling for a direct perception account, i.e. motion-from-motion in all cases. Moreover they leave open the alternative for representational formats: motion or shape from a non optimal, or even non recognisable, shape put in motion. So shape remains a possible format for static as well as for dynamic stimuli, at least for a set of category features of speech sounds, in our case the main visible vowel gesture, rounding.KeywordsAnticipationface angle viewrounding feature/gestureshape-frommotionshape-from-shading

A single pool of untrained subjects was tested for interactions across two bimodal perception conditions: audio-tactile, in which subjects heard and felt speech, and visual-tactile, in which subjects saw and felt speech. Identifications of English obstruent consonants were compared in bimodal and no-tactile baseline conditions. Results indicate that tactile information enhances speech perception by about 10 percent, regardless of which other mode (auditory or visual) is active. However, within-subject analysis indicates that individual subjects who benefit more from tactile information in one cross-modal condition tend to benefit less from tactile information in the other.

Cognitive Processing in Bilinguals

High-density electrocorticography reveals sensorimotor cortex engagement in two distinct sites with different roles during audiovisual, audio, and visual speech perception.

Research has shown that visual speech perception can assist accuracy in identification of spoken words. However, little is known about the dynamics of the processing mechanisms involved in audiovisual integration. In particular, architecture and capacity, measured using response time methodologies, have not been investigated. An issue related to architecture concerns whether the auditory and visual sources of the speech signal are integrated “early” or “late.” We propose that “early” integration most naturally corresponds to coactive processing whereas “late” integration corresponds to separate decisions parallel processing. We implemented the double factorial paradigm in two studies. First, we carried out a pilot study using a two-alternative forced-choice discrimination task to assess architecture, decision rule, and provide a preliminary assessment of capacity (integration efficiency). Next, Experiment 1 was designed to specifically assess audiovisual integration efficiency in an ecologically valid way by including lower auditory S/N ratios and a larger response set size. Results from the pilot study support a separate decisions parallel, late integration model. Results from both studies showed that capacity was severely limited for high auditory signal-to-noise ratios. However, Experiment 1 demonstrated that capacity improved as the auditory signal became more degraded. This evidence strongly suggests that integration efficiency is vitally affected by the S/N ratio.

Purpose: The present study was designed to investigate the neural correlates of German Sign Language (Deutsche Gebärdensprache; DGS) processing. In particular, was expected the impact of the visuo-spatial mode in sign language on underlying neural networks compared to the impact of the interpretation of linguistic information. Methods: For this purpose, two groups of participants took part in a functional MRI study at 3 Tesla. One group consisted of prelingually deafened users of DGS, the other group of hearing non-signers naïve to sign language. The two groups were presented with identical video sequences comprising DGS sentences in form of dialoges. To account for substantial interindividual anatomical variability observed in the group of deaf participants, the brain responses in the two groups of subjects were analyzed with two different procedures. Results: Results from a multi-subject averaging approach were contrasted with an analysis, which can account for the considerable inter-individual variability of gross anatomical landmarks. The anatomy-based approach indicated that individuals’ responses to proper DGS processing was tied up with a leftward asymmetry in the dorsolateral prefrontal cortex, anterior and middle temporal gyrus, and visual association cortices. In contrast, standard multi-subject averaging of deaf individuals during DGS perception revealed a less lateralized peri- and extrasylvian network. Furthermore, voxel-based analyses of the brains’ morphometry evidenced a white-matter deficit in the left posterior longitudinal and inferior uncinate fasciculi and a steeper slope of the posterior part of the left Sylvian Fissure (SF) in the deaf individuals. Conclusion: These findings may imply that the cerebral anatomy of deaf individuals has undergone structural changes as a function of monomodal visual sign language perception during childhood and adolescence.

Although by 370 b.c. Hippocrates had concluded that the brain played a role in the generation of behavior (Carlson, 2003), it was not until seminal studies performed in the 1860s that specific human cognitive abilities began to be mapped onto regions of the brain with some accuracy. Postmortem studies by Paul Broca and Karl Wernicke allowed individuals' cognitive deficits to be linked tentatively to lesions discovered in specific brain regions. It was not until almost a century later that the advent of noninvasive neuroimaging techniques allowed the functional neuroanatomy of the brain of a living individual to be visualized for the first time. The striking contemporary images produced by functional magnetic resonance imaging (fMRI) have allowed researchers to examine in detail the neural correlates of a wide range of cognitive processes such as visual perception, speech perception, and short-term memory.

In response to the ongoing sex data gap, the present study provides a qualitative exploration of females’ nutritional experiences in elite sporting environments. Semistructured interviews were conducted with multiple participant groups (n = 18), including athletes (n = 7), practitioners (n = 6), and researchers (n = 5) across differing disciplines within professional sporting organizations and/or national governing bodies. Combined content and thematic analysis provided an insight into the specific factors influencing current sport nutrition practices. A common theme highlighted among all participant groups was the paradoxical struggle between adequate fueling for training and competition demands, and the fear this may impact body mass and body composition goals. This tension was identified as being rooted within athletes’ perceptions of body image and driven by other participant groups and wider societal ideals. Each participant group also highlighted influences on cravings and approaches to food and dietary supplementation, centered around individual perceptions and challenges driven by symptomology associated with the female menstrual cycle and contraceptive use. To address these challenges, all participant groups called for more research to inform future change and continuing education pathways. In summary, this study contributes to providing a more complete understanding of elite female athlete sport nutrition experiences than currently exists. Multiple perspectives highlight the complexity of providing sport nutrition support to elite female athlete populations and directs future research, and practice, to reconsider one-size-fits-all approaches and acknowledge unique individual contexts which may influence these areas.