An Assessment of Behavioral Dynamic Information Processing Measures in Audiovisual Speech Perception

Introduction: To the best of our knowledge, there is a lack of reliable, validated, and standardized (Dutch) measuring instruments to document visual speech perception in a structured way. This study aimed to: (1) evaluate the effects of age, gender, and the used word list on visual speech perception examined by a first version of the Dutch Test for (Audio-)Visual Speech Perception on word level (TAUVIS-words) and (2) assess the internal reliability of the TAUVIS-words. Methods: Thirty-nine normal-hearing adults divided into the following 3 age categories were included: (1) younger adults, age 18–39 years; (2) middle-aged adults, age 40–59 years; and (3) older adults, age >60 years. The TAUVIS-words consist of 4 word lists, i.e., 2 monosyllabic word lists (MS 1 and MS 2) and 2 polysyllabic word lists (PS 1 and PS 2). A first exploration of the effects of age, gender, and test stimuli (i.e., the used word list) on visual speech perception was conducted using the TAUVIS-words. A mixed-design analysis of variance (ANOVA) was conducted to analyze the results statistically. Lastly, the internal reliability of the TAUVIS-words was assessed by calculating the Chronbach α. Results: The results revealed a significant effect of the used list. More specifically, the score for MS 1 was significantly better compared to that for PS 2, and the score for PS 1 was significantly better compared to that for PS 2. Furthermore, a significant main effect of gender was found. Women scored significantly better compared to men. The effect of age was not significant. The TAUVIS-word lists were found to have good internal reliability. Conclusion: This study was a first exploration of the effects of age, gender, and test stimuli on visual speech perception using the TAUVIS-words. Further research is necessary to optimize and validate the TAUVIS-words, making use of a larger study sample.

Virtual personal assistants (VPAs) like Siri and Alexa have become common objects in households. Users frequently rely on these systems to search the internet or help retrieve information. As such, it is important to know how using these products affect cognitive processes like memory. Previous research suggests that visual speech perception influences auditory perception in human-human interactions. However, many of these VPAs are designed as a box or sphere that does not interact with the user visually. This lack of visual speech perception when interacting with a VPA could affect the human interaction with a system and their retention of information such as determining how many ounces are in a cup or how to greet someone in another language. This poses the question of whether the design of these VPAs is preventing the ability of users to retain the information they get from these systems. To test this, we designed an experiment that will explore interactions between user memory and either a traditional audio presentation (as is found with Siri or Alexa, for example) or one that allows for visual speech perception. Participants were asked to listen to an audio clip of a nonsensical story. In one condition, participants were asked to listen while looking at a blank screen (analogous to the lack of visual feedback inherent when working with current VPA designs). After a block of 25 audio clips, the participants took a test on the information heard. This process was repeated with an animated face with synchronized mouth movements instead of a black screen. Other participants will experience the same two presentations, but in reverse order as to counterbalance condition presentation. Data collection is currently underway. We predicted that VPA paired with synchronized lip movement would promote visual speech perception and thus help participants retain information. While we are still collecting data, the trend currently does not show a significant difference between audio and lip movement conditions. This could be an indication of differing abilities in lipreading.

Background: Pathological and physiological disabilities of human auditory receptors reduce speech intelligibility under hearing loss (HL) at high frequencies. Objectives: The objective of the study is to investigate the speech-intelligibility-enhancement (SIE) in elderly with high-frequency HL (HFHL) through visual-speech-perception (VSP). The hypothesis is that the VSP of spondee words (two-syllable words with equal stress on both syllables) is properly recognized and reproduced with HFHL. Also; to decide the statistical relevance of the precise significant difference in speech discrimination (SD) with VSP on the subjected ear. Methods: Observational and descriptive studies of SIE with VSP on 12 elderly listeners (24 clear ears; without wax impaction) with progressive and bilateral HFHL were examined. The entire experimental records were checked for distribution with normal (Gaussian) using the Shapiro–Wilk's and paired Student's t-test (parametric test) had the 5% (0.05) significance level (α). Results: We analyzed, improved, and better performance in SIE with HFHL through VSP. The statistical P (probability) measure was calculated nothing, i.e., 0 (P < 0.05). Therefore, investigators strongly and carefully discarded the null hypothesis consideration. There was some significant value of statistical variation also exists with VSP. Moreover, with the help of the confidence-interval analysis, the same conclusion was achieved. Conclusions: In this study, it is concluded that the elderly-individuals of this experiment, whose mother tongue and official language is Hindi and English, respectively, they demonstrate the SIE with HFHL through VSP. This investigation also helps to improve the hearing sensitivity to some extent with VSP.

Cognitive Processing in Bilinguals

s International. 40/09B, 4543 (University Microfilms No. 8004002), 1979. 13. Momtahan KL. Mapping of psychoacoustic parameters to the perceived urgency of auditory warning signals [dissertation] 14. Hellier E, Edworthy J. On using psychophysical techniques to achieve urgency mapping in auditory warnings. Applied Ergonomics, 30, 167-171, 1999. 15. Brewster SA, Wright PC, Edwards A. Experimentally Derived Guidelines for the Creation of Earcons, in Proceedings of HCI 95, Huddersfield, UK, pp. 155-159, 1995. 16. McGookin DK, Brewster SA. An investigation into the identification of concurrently presented earcons. Proceedings of the 2003 International Conference on Auditory Display, Boston, MA, USA,

The role of visual spatial attention in audiovisual speech perception

The study explored whether visual information improves speech identification in typically developing children with normal hearing when the auditory signal is spectrally degraded. Children (n=69) and adults (n=15) were presented with noise-vocoded sentences from the Children's Co-ordinate Response Measure (Rosen, 2011) in auditory-only or audiovisual conditions. The number of bands was adaptively varied to modulate the degradation of the auditory signal, with the number of bands required for approximately 79% correct identification calculated as the threshold. The youngest children (4- to 5-year-olds) did not benefit from accompanying visual information, in comparison to 6- to 11-year-old children and adults. Audiovisual gain also increased with age in the child sample. The current data suggest that children younger than 6 years of age do not fully utilize visual speech cues to enhance speech perception when the auditory signal is degraded. This evidence not only has implications for understanding the development of speech perception skills in children with normal hearing but may also inform the development of new treatment and intervention strategies that aim to remediate speech perception difficulties in pediatric cochlear implant users.

Perception based method for the investigation of audiovisual integration of speech

Multisensory events in our natural environment unfold at multiple temporal scales over extended periods of time. This functional magnetic resonance imaging study investigated whether the brain uses transient (onset, offset) or sustained temporal codes to effectively integrate incoming visual and auditory signals within the cortical hierarchy. Subjects were presented with 1) velocity-modulated radial motion, 2) amplitude-modulated sound, or 3) an in phase combination of both in blocks of variable durations to dissociate transient and sustained blood oxygen level-dependent responses. Audiovisual interactions emerged primarily for transient onset and offset responses highlighting the importance of rapid stimulus transitions for multisensory integration. Strikingly, audiovisual interactions for onset and offset transients were dissociable at the functional and anatomical level. Low-level sensory areas integrated audiovisual inputs at stimulus onset in a superadditive fashion to enhance stimulus salience. In contrast, higher order association areas showed subadditive integration profiles at stimulus offset possibly reflecting the formation of higher order representations. In conclusion, multisensory integration emerges at multiple levels of the cortical hierarchy using different temporal codes and integration profiles. From a methodological perspective, these results highlight the limitations of conventional event related or block designs that cannot characterize these rich dynamics of audiovisual integration.

<p>In four experiments, we examined whether facial expressions used while singing carry musical information that can be “read” by viewers. In Experiment 1, participants saw silent video recordings of sung melodic intervals and judged the size of the interval they imagined the performers to be singing. Participants discriminated interval sizes on the basis of facial expression and discriminated large from small intervals when only head movements were visible. Experiments 2 and 3 confirmed that facial expressions influenced judgments even when the auditory signal was available. When matched with the facial expressions used to perform a large interval, audio recordings of sung intervals were judged as being larger than when matched with the facial expressions used to perform a small interval. The effect was not diminished when a secondary task was introduced, suggesting that audio-visual integration is not dependent on attention. Experiment 4 confirmed that the secondary task reduced participants’ ability to make judgments that require conscious attention. The results provide the first evidence that facial expressions influence perceived pitch relations.</p>

Recent studies have questioned past conclusions regarding the mechanisms of the McGurk illusion, especially how McGurk susceptibility might inform our understanding of audiovisual (AV) integration. We previously proposed that the McGurk illusion is likely attributable to a default mechanism, whereby either the visual system, auditory system, or both default to specific phonemes—those implicated in the McGurk illusion. We hypothesized that the default mechanism occurs because visual stimuli with an indiscernible place of articulation (like those traditionally used in the McGurk illusion) lead to an ambiguous perceptual environment and thus a failure in AV integration. In the current study, we tested the default hypothesis as it pertains to the auditory system. Participants performed two tasks. One task was a typical McGurk illusion task, in which individuals listened to auditory-/ba/ paired with visual-/ga/ and judged what they heard. The second task was an auditory-only task, in which individuals transcribed trisyllabic words with a phoneme replaced by silence. We found that individuals’ transcription of missing phonemes often defaulted to ‘/d/t/th/’, the same phonemes often experienced during the McGurk illusion. Importantly, individuals’ default rate was positively correlated with their McGurk rate. We conclude that the McGurk illusion arises when people fail to integrate visual percepts with auditory percepts, due to visual ambiguity, thus leading the auditory system to default to phonemes often implicated in the McGurk illusion.

The present study was designed to examine age differences in the ability to use voice information acquired intentionally (Experiment 1) or incidentally (Experiment 2) as an aid to spoken word identification. Following both implicit and explicit voice learning, participants were asked to identify novel words spoken either by familiar talkers (ones they had been exposed to in the training phase) or by 4 unfamiliar voices. In both experiments, explicit memory for talkers' voices was significantly lower in older than in young listeners. Despite this age-related decline in voice recognition, however, older adults exhibited equivalent, and in some cases greater, benefit than young listeners from having words spoken by familiar talkers. Implications of the findings for age-related changes in explicit versus implicit memory systems are discussed.

Previous studies have found that perception in older people benefits from multisensory over unisensory information. As normal speech recognition is affected by both the auditory input and the visual lip movements of the speaker, we investigated the efficiency of audio and visual integration in an older population by manipulating the relative reliability of the auditory and visual information in speech. We also investigated the role of the semantic context of the sentence to assess whether audio–visual integration is affected by top-down semantic processing. We presented participants with audio–visual sentences in which the visual component was either blurred or not blurred. We found that there was a greater cost in recall performance for semantically meaningless speech in the audio–visual ‘blur’ compared to audio–visual ‘no blur’ condition and this effect was specific to the older group. Our findings have implications for understanding how aging affects efficient multisensory integration for the perception of speech and suggests that multisensory inputs may benefit speech perception in older adults when the semantic content of the speech is unpredictable.

It has been proposed that the common metric between audio and visual speech perception could be just kinematics (say dynamics), a somewhat rather old gibsonian proposal illustrated until recently by moving point-lights displays. Results reported here address the same question in the frame of another paradigm than moving dots. Full images of the face were presented in moving and static conditions along a naturally silent vowel-to-vowel rounding gesture (our “silent pause lipreading paradigm”). In order to compare the benefit of motion vs. shape, the change in angle paradigm for face perception was used for feature/gesture identification. The question is then: Does viewer-centred optimisation of the projection of the rounding gesture override motion benefit? Such a saturation effect found for profiles supports a shape-from-shading processing for front views, enhanced by the shape-from-motion system, such systems being recruited in case of undersampling (point-lights) or for out of shape projections. These computational approaches avoid calling for a direct perception account, i.e. motion-from-motion in all cases. Moreover they leave open the alternative for representational formats: motion or shape from a non optimal, or even non recognisable, shape put in motion. So shape remains a possible format for static as well as for dynamic stimuli, at least for a set of category features of speech sounds, in our case the main visible vowel gesture, rounding.KeywordsAnticipationface angle viewrounding feature/gestureshape-frommotionshape-from-shading

Common-coding theory posits that (1) perceiving an action activates the same representations of motor plans that are activated by actually performing that action, and (2) because of individual differences in the ways that actions are performed, observing recordings of one's own previous behavior activates motor plans to an even greater degree than does observing someone else's behavior. We hypothesized that if observing oneself activates motor plans to a greater degree than does observing others, and if these activated plans contribute to perception, then people should be able to lipread silent video clips of their own previous utterances more accurately than they can lipread video clips of other talkers. As predicted, two groups of participants were able to lipread video clips of themselves, recorded more than two weeks earlier, significantly more accurately than video clips of others. These results suggest that visual input activates speech motor activity that links to word representations in the mental lexicon.