GYMNASTS UTILIZE VISUAL AND AUDITORY INFORMATION FOR BEHAVIOURAL SYNCHRONIZATION IN TRAMPOLINING

Although speech perception is usually considered an auditory process, studies have shown that visual information provided by the movements of a talker’s mouth and face strongly influences what an observer perceives even when the auditory signal is clear and unambiguous. The now classic “McGurk effect” is an excellent example: the auditory syllable /bi/ presented in synchrony with a videotape of a talker saying the syllable /gi/ is usually perceived as /di/, a syllable not presented to either modality. This phenomenon has important implications for theories of speech perception which must account for how and why auditory and visual signals are integrated during phonetic processing. The purpose of the current talk is to describe some of the research that has contributed to our understanding of how visual and auditory information are used in phonetic perception. Specifically, this paper focuses on three issues: (1) what factors influence the integration of auditory and visual information; (2) the nature of the auditory and visual information used in phonetic perception; and (3) the interaction of the two sources of information during phonetic processing. The implications of these findings for theories of speech perception are discussed.

One of the requirements for being a successful musical conductor is to be able to locate sounds instantaneously in time and space. Because this requires the integration of auditory and visual information, the purpose of this study was to examine multisensory processing in conductors and a matched set of control subjects. Subjects participated in a series of behavioral tasks, including pitch discrimination, temporal-order judgment (TOJ), and target localization. Additionally, fMRI scans were done on a subset of subjects who performed a multisensory TOJ task. Analyses of behavioral data indicate that, in the auditory realm, conductors were more accurate in both pitch discrimination and TOJs as well as in locating targets in space. Furthermore, these same subjects also demonstrated a benefit from the combination of auditory and visual information that was not observed in control subjects when locating visual targets. Finally, neural substrates in BA 37, 39/40 were identified as potential areas underlying the conductors' superior multisensory TOJs. Data collection and analyses are ongoing and will lead to an improved understanding of multisensory integration in a complex, musical behavior.

Most studies have been unable to identify reliable acoustic cues for the recognition of the English nonsibilant fricatives [see text]. The present study was designed to test the extent to which the perception of these fricatives by normal-hearing adults is based on other sources of information, namely, linguistic context and visual information. In Experiment 1, target words beginning with /f/, /theta/, /s/, or [see text] were preceded by either a semantically congruous or incongruous precursor sentence. Results showed an effect of linguistic context on the perception of the distinction between /f/ and /theta/ and on the acoustically more robust distinction between /s/ and [see text]. In Experiment 2, participants identified syllables consisting of the fricatives [see text] paired with the vowels /i, a, u/. Three conditions were contrasted: Stimuli were presented with (a) both auditory and visual information, (b) auditory information alone, or (c) visual information alone. When errors in terms of voicing were ignored in all 3 conditions, results indicated that perception of these fricatives is as good with visual information alone as with both auditory and visual information combined, and better than for auditory information alone. These findings suggest that accurate perception of nonsibilant fricatives derives from a combination of acoustic, linguistic, and visual information.

Recognizing fencing attacks from auditory and visual information: A comparison between expert fencers and novices

In this chapter, the authors examine the extent to which preschool children use auditory information in drawing inductive inferences about animals. In a study by J. T. H. Wong, 4-year-old children's use of auditory and visual information has been compared. A series of studies by S. A. Gelman have shown that children as young as 3 years of age were able to go beyond perceptual cues and use conceptual information, such as category labels, to make judgments. Preschool children tended to use motion information more often than category labels to make categorical judgments. Children were presented with 2 pairs of animal stimuli, one pair at a time. The children were first taught a new property about the target and were then asked to infer if the property was also true for the test animal. The authors do believe that information of shape, sound and labels interact to guide children's categorical judgment.

Objectives: The present study compared performance on an auditory word recognition task with different visual, auditory, and contextual information across age groups to determine how these variables affect auditory comprehension in normal aging. Methods: We conducted an auditory word recognition task with 20 normal adults from each of the following age groups: young, middle-aged, old, and elderly, and compared their correct response scores and analyzed their error responses. We also conducted a correlation analysis between the score of auditory comprehension in Reading Diagnostic Assessment, RDA and the performance of the auditory word recognition task. Results: There was a significant between-group main effect, as well as within-group main effects of auditory and contextual information, and interaction effects between auditory information and group, contextual information and group, and auditory information and contextual information. The old and elderly groups made more errors in the following order of error types: unrelated errors, formal errors, mixed errors, and semantic errors; and there were no significant differences in error patterns between the two groups. There was a significant positive correlation between the RDA and the auditory word recognition task. Conclusion: This study confirms that auditory word recognition performance declines with aging in normal adults, and that difficulties may be more pronounced under certain conditions, such as noise and low contextual sentences.

In this study, the influences of environmentally friendly visual and auditory information on traffic noise perception were surveyed by the using electroencephalogram. Green rural region image and CBD (central business district) image in urban city were used as visual informations. And traffic noise, signal and environmental music were used to detect the impact on electroencephalogram variance. It was revealed that green rural region image caused <TEX>${\alpha}-wave$</TEX> ratio increase about 10% and environmental music increased <TEX>${\alpha}-wave$</TEX> ratio approximately <TEX>$40{\sim}50%$</TEX>. The results of this study improved that environmentally friendly visual and auditory information had an effect on decreasing traffic noise loudness to some extents.

A human acting and moving in the environment often produces sounds, which are related to his or her body movements. Movement control requires the continuous use of perceptual information about the ongoing movement. It was investigated whether humans can and do use sound produced by their movements as auditory information for the control of their movements. In the experiments the movements or tasks required from the subjects were varied, as well as the available auditory and visual information. The success in the movement tasks was measured. The results obtained so far show that auditory information is used for movement control by the subjects, especially with complicated movements.

The "McGurk effect" demonstrates that visual (lip-read) information is used during speech perception even when it is discrepant with auditory information. While this has been established as a robust effect in subjects from Western cultures, our own earlier results had suggested that Japanese subjects use visual information much less than American subjects do (Sekiyama & Tohkura, 1993). The present study examined whether Chinese subjects would also show a reduced McGurk effect due to their cultural similarities with the Japanese. The subjects were 14 native speakers of Chinese living in Japan. Stimuli consisted of 10 syllable (/ba/, /pa/, /ma/, /wa/, /da/, /ta/, /na/, /ga/, /ka/, /ra/) pronounced by two speakers, one Japanese and one American. Each auditory syllable was dubbed onto every, visual syllable within one speaker, resulting in 100 audiovisual stimuli in each language. The subjects' main task was to report what they thought they had heard while looking at and listening to the speaker while the stimuli were being uttered. Compared with previous results obtained with American subjects, the Chinese subjects showed a weaker McGurk effect. The results also showed that the magnitude of the McGurk effect depends on the length of time the Chinese subjects had lived in Japan. Factors that foster and alter the Chinese subjects' reliance on auditory information are discussed.

Speech perception under natural conditions entails integration of auditory and visual information. Understanding how visual and auditory speech information are integrated requires detailed descriptions of the nature and processing of visual speech information. To understand better the process of gathering visual information, we studied the distribution of face-directed fixations of humans performing an audiovisual speech perception task to characterise the degree of asymmetrical viewing and its relationship to speech intelligibility. Participants showed stronger gaze fixation asymmetries while viewing dynamic faces, compared to static faces or face-like objects, especially when gaze was directed to the talkers' eyes. Although speech perception accuracy was significantly enhanced by the viewing of congruent, dynamic faces, we found no correlation between task performance and gaze fixation asymmetry. Most participants preferentially fixated the right side of the faces and their preferences persisted while viewing horizontally mirrored stimuli, different talkers, or static faces. These results suggest that the asymmetrical distributions of gaze fixations reflect the participants' viewing preferences, rather than being a product of asymmetrical faces, but that this behavioural bias does not predict correct audiovisual speech perception.

Observers' accuracy in using time-to-arrival (Ta) information was examined in 4 experiments. The issues included use of visual vs. acoustic Ta information, use of acoustic Ta information by blind Ss, use of Ta information controlling for velocity, and effects of angle of approach and arrival time on judgment accuracy. Visual information was used more efficiently than audiovisual and auditory information. Blind Ss used acoustical approach information as accurately as sighted Ss used visual information. Radial, oblique, and transverse orientations were used to examine effects of approach trajectory. Radial events were underestimated, whereas the more accurate transverse approach was likely to be overestimated. Oblique angle events yielded intermediate accuracies implying a spatiotemporal anisotropy. Women underestimated Ta more than did men. Possible reasons for Ss' judgment accuracy, including linear vs. nonlinear optical changes and relation to spatial skills and experience, were discussed.

High-density electrocorticography reveals sensorimotor cortex engagement in two distinct sites with different roles during audiovisual, audio, and visual speech perception.

Effective rendering technique of auditory information for an augmented reality (AR) device has been investigated. Researchers and industries are searching for new applications incorporating AR and Virtual Reality (VR) technologies, which provide enhanced user-experience of interactivity and intimacy. To utilize these benefits best, homogeneous integration between visual and auditory information is important. To date, the binaural technology based on head-related transfer function (HRTF) has been used to create immersive and three-dimensional audio objects for VR and AR devices. In this study, we compared the HRTF method with a stereophonic panning method that only controls Inter-aural Level Difference (ILD) for an AR device (a smart see-through glass) and investigated the precision of auditory information required for a coherent representation with a target visual image at the locations of 0 degree, -5 degrees, and -10 degrees in a counter clockwise. Auditory stimuli were rendered to have target locations in the horizontal plane from + 45 degrees to -45 degrees with a 5-degree interval. In the subjective evaluation, each participant answered whether a randomly given auditory location was matched to a target location of the visual image. The results showed that two audio rendering methods did not produce significant difference in creating integrated perception in an AR device.

Human beings are doing a smooth communication with the auditory, visual and haptic information. Multimodal interfaces are utilized the communication operation assist could be improved by visual and auditory information given the time difference in presentation. In our previous research, the validity of multimodal assists with time difference were experimented by vibration control task for a pendulum. The experimental results showed that giving a time difference to the information is effective in the machine operation performance. However, the results depended on the task. Therefore, the experiment is conducted by drive simulation, as a more general task. And in this paper, validity of proposed assist with the tasks is verified, as a first step, the effect on the operating characteristics of each assist. Finally, experimental results show that giving the visual assist is effective in reducing the angle error, and the auditory assist is effective in reducing the position and angle error.

Most research on ethnicity in neuroscience and social psychology has focused on visual cues. However, accents are central social markers of ethnicity and strongly influence evaluations of others. Here, we examine how varying auditory (vocal accent) and visual (facial appearance) information about others affects neural correlates of ethnicity-related expectancy violations. Participants listened to standard German and Turkish-accented speakers and were subsequently presented with faces whose ethnic appearance was either congruent or incongruent to these voices. We expected that incongruent targets (e.g. German accent/Turkish face) would be paralleled by a more negative N2 event-related brain potential (ERP) component. Results confirmed this, suggesting that incongruence was related to more effortful processing of both Turkish and German target faces. These targets were also subjectively judged as surprising. Additionally, varying lateralization of ERP responses for Turkish and German faces suggests that the underlying neural generators differ, potentially reflecting different emotional reactions to these targets. Behavioral responses showed an effect of violated expectations: German-accented Turkish-looking targets were evaluated as most competent of all targets. We suggest that bringing together neural and behavioral measures of expectancy violations, and using both visual and auditory information, yields a more complete picture of the processes underlying impression formation.