Recognizing fencing attacks from auditory and visual information: A comparison between expert fencers and novices

In this chapter, the authors examine the extent to which preschool children use auditory information in drawing inductive inferences about animals. In a study by J. T. H. Wong, 4-year-old children's use of auditory and visual information has been compared. A series of studies by S. A. Gelman have shown that children as young as 3 years of age were able to go beyond perceptual cues and use conceptual information, such as category labels, to make judgments. Preschool children tended to use motion information more often than category labels to make categorical judgments. Children were presented with 2 pairs of animal stimuli, one pair at a time. The children were first taught a new property about the target and were then asked to infer if the property was also true for the test animal. The authors do believe that information of shape, sound and labels interact to guide children's categorical judgment.

In synchronized trampolining, two gymnasts perform the same routine at the same time. While trained gymnasts are thought to coordinate their own movements with the movements of another gymnast by detecting relevant movement information, the question arises how visual and auditory information contribute to the emergence of synchronicity between both gymnasts. Therefore the aim of this study was to examine the role of visual and auditory information in the emergence of coordinated behaviour in synchronized trampolining. Twenty female gymnasts were asked to synchronize their leaps with the leaps of a model gymnast, while visual and auditory information was manipulated. The results revealed that gymnasts needed more leaps to reach synchronicity when only either auditory (12.9 leaps) or visual information (10.8 leaps) was available, as compared to when both auditory and visual information was available (8.1 leaps). It is concluded that visual and auditory information play significant roles in synchronized trampolining, whilst visual information seems to be the dominant source for emerging behavioural synchronization, and auditory information supports this emergence.

A human acting and moving in the environment often produces sounds, which are related to his or her body movements. Movement control requires the continuous use of perceptual information about the ongoing movement. It was investigated whether humans can and do use sound produced by their movements as auditory information for the control of their movements. In the experiments the movements or tasks required from the subjects were varied, as well as the available auditory and visual information. The success in the movement tasks was measured. The results obtained so far show that auditory information is used for movement control by the subjects, especially with complicated movements.

Although speech perception is usually considered an auditory process, studies have shown that visual information provided by the movements of a talker’s mouth and face strongly influences what an observer perceives even when the auditory signal is clear and unambiguous. The now classic “McGurk effect” is an excellent example: the auditory syllable /bi/ presented in synchrony with a videotape of a talker saying the syllable /gi/ is usually perceived as /di/, a syllable not presented to either modality. This phenomenon has important implications for theories of speech perception which must account for how and why auditory and visual signals are integrated during phonetic processing. The purpose of the current talk is to describe some of the research that has contributed to our understanding of how visual and auditory information are used in phonetic perception. Specifically, this paper focuses on three issues: (1) what factors influence the integration of auditory and visual information; (2) the nature of the auditory and visual information used in phonetic perception; and (3) the interaction of the two sources of information during phonetic processing. The implications of these findings for theories of speech perception are discussed.

One of the requirements for being a successful musical conductor is to be able to locate sounds instantaneously in time and space. Because this requires the integration of auditory and visual information, the purpose of this study was to examine multisensory processing in conductors and a matched set of control subjects. Subjects participated in a series of behavioral tasks, including pitch discrimination, temporal-order judgment (TOJ), and target localization. Additionally, fMRI scans were done on a subset of subjects who performed a multisensory TOJ task. Analyses of behavioral data indicate that, in the auditory realm, conductors were more accurate in both pitch discrimination and TOJs as well as in locating targets in space. Furthermore, these same subjects also demonstrated a benefit from the combination of auditory and visual information that was not observed in control subjects when locating visual targets. Finally, neural substrates in BA 37, 39/40 were identified as potential areas underlying the conductors' superior multisensory TOJs. Data collection and analyses are ongoing and will lead to an improved understanding of multisensory integration in a complex, musical behavior.

Objectives: The present study compared performance on an auditory word recognition task with different visual, auditory, and contextual information across age groups to determine how these variables affect auditory comprehension in normal aging. Methods: We conducted an auditory word recognition task with 20 normal adults from each of the following age groups: young, middle-aged, old, and elderly, and compared their correct response scores and analyzed their error responses. We also conducted a correlation analysis between the score of auditory comprehension in Reading Diagnostic Assessment, RDA and the performance of the auditory word recognition task. Results: There was a significant between-group main effect, as well as within-group main effects of auditory and contextual information, and interaction effects between auditory information and group, contextual information and group, and auditory information and contextual information. The old and elderly groups made more errors in the following order of error types: unrelated errors, formal errors, mixed errors, and semantic errors; and there were no significant differences in error patterns between the two groups. There was a significant positive correlation between the RDA and the auditory word recognition task. Conclusion: This study confirms that auditory word recognition performance declines with aging in normal adults, and that difficulties may be more pronounced under certain conditions, such as noise and low contextual sentences.

Most studies have been unable to identify reliable acoustic cues for the recognition of the English nonsibilant fricatives [see text]. The present study was designed to test the extent to which the perception of these fricatives by normal-hearing adults is based on other sources of information, namely, linguistic context and visual information. In Experiment 1, target words beginning with /f/, /theta/, /s/, or [see text] were preceded by either a semantically congruous or incongruous precursor sentence. Results showed an effect of linguistic context on the perception of the distinction between /f/ and /theta/ and on the acoustically more robust distinction between /s/ and [see text]. In Experiment 2, participants identified syllables consisting of the fricatives [see text] paired with the vowels /i, a, u/. Three conditions were contrasted: Stimuli were presented with (a) both auditory and visual information, (b) auditory information alone, or (c) visual information alone. When errors in terms of voicing were ignored in all 3 conditions, results indicated that perception of these fricatives is as good with visual information alone as with both auditory and visual information combined, and better than for auditory information alone. These findings suggest that accurate perception of nonsibilant fricatives derives from a combination of acoustic, linguistic, and visual information.

Developments of head-coupled control/display systems have focused primarily on the display of three dimensional visual information, as the visual system is the optimal sensory channel for the aquisition of spatial information in humans. The auditory system improves the efficiency of vision, however, by obtaining spatial information about relevant objects outside of the visual field of view. This auditory information can be used to direct head and eye movements. Head-coupled display systems, can also benefit from the addition of auditory spatial information, as it provides a natural method of signaling the location of important events outside of the visual field of view. This symposium will report on current efforts in the developments of head-coupled display systems, with an emphasis on the auditory spatial component. The first paper “Virtual Interface Environment Workstations”, by Scott S. Fisher, will report on the development of a prototype virtual environment. This environment consists of a head-mounted, wide-angle, stereoscopic display system which is controlled by operator position, voice, and gesture. With this interface, an operator can virtually explore a 360 degree synthesized environment, and viscerally interact with its components. The second paper, “A Virtual Display System For Conveying Three-Dimensional Acoustic Information” by Elizabeth M. Wenzel, Frederic L. Wightman and Scott H. Foster, will report on the development of a method of synthetically generating three-dimensional sound cues for the above-mentioned interface. The development of simulated auditory spatial cues is limited to some extent, by our knowlege of auditory spatial processing. The remaining papers will report on two areas of auditory space perception that have recieved little attention until recently. “Perception of Real and Simulated Motion in the Auditory Modality”, by Thomas Z. Strybel, will review recent research on auditory motion perception, because a natural acoustic environment must contain moving sounds. This review will consider applications of this knowledge to head-coupled display systems. The last paper, “Auditory Psychomotor Coordination”, will examine the interplay between the auditory, visual and motor systems. The specific emphasis of this paper is the use of auditory spatial information in the regulation of motor responses so as to provide efficient application of the visual channel.

In this study, the influences of environmentally friendly visual and auditory information on traffic noise perception were surveyed by the using electroencephalogram. Green rural region image and CBD (central business district) image in urban city were used as visual informations. And traffic noise, signal and environmental music were used to detect the impact on electroencephalogram variance. It was revealed that green rural region image caused <TEX>${\alpha}-wave$</TEX> ratio increase about 10% and environmental music increased <TEX>${\alpha}-wave$</TEX> ratio approximately <TEX>$40{\sim}50%$</TEX>. The results of this study improved that environmentally friendly visual and auditory information had an effect on decreasing traffic noise loudness to some extents.

Effective rendering technique of auditory information for an augmented reality (AR) device has been investigated. Researchers and industries are searching for new applications incorporating AR and Virtual Reality (VR) technologies, which provide enhanced user-experience of interactivity and intimacy. To utilize these benefits best, homogeneous integration between visual and auditory information is important. To date, the binaural technology based on head-related transfer function (HRTF) has been used to create immersive and three-dimensional audio objects for VR and AR devices. In this study, we compared the HRTF method with a stereophonic panning method that only controls Inter-aural Level Difference (ILD) for an AR device (a smart see-through glass) and investigated the precision of auditory information required for a coherent representation with a target visual image at the locations of 0 degree, -5 degrees, and -10 degrees in a counter clockwise. Auditory stimuli were rendered to have target locations in the horizontal plane from + 45 degrees to -45 degrees with a 5-degree interval. In the subjective evaluation, each participant answered whether a randomly given auditory location was matched to a target location of the visual image. The results showed that two audio rendering methods did not produce significant difference in creating integrated perception in an AR device.

Human beings are doing a smooth communication with the auditory, visual and haptic information. Multimodal interfaces are utilized the communication operation assist could be improved by visual and auditory information given the time difference in presentation. In our previous research, the validity of multimodal assists with time difference were experimented by vibration control task for a pendulum. The experimental results showed that giving a time difference to the information is effective in the machine operation performance. However, the results depended on the task. Therefore, the experiment is conducted by drive simulation, as a more general task. And in this paper, validity of proposed assist with the tasks is verified, as a first step, the effect on the operating characteristics of each assist. Finally, experimental results show that giving the visual assist is effective in reducing the angle error, and the auditory assist is effective in reducing the position and angle error.

Most research on ethnicity in neuroscience and social psychology has focused on visual cues. However, accents are central social markers of ethnicity and strongly influence evaluations of others. Here, we examine how varying auditory (vocal accent) and visual (facial appearance) information about others affects neural correlates of ethnicity-related expectancy violations. Participants listened to standard German and Turkish-accented speakers and were subsequently presented with faces whose ethnic appearance was either congruent or incongruent to these voices. We expected that incongruent targets (e.g. German accent/Turkish face) would be paralleled by a more negative N2 event-related brain potential (ERP) component. Results confirmed this, suggesting that incongruence was related to more effortful processing of both Turkish and German target faces. These targets were also subjectively judged as surprising. Additionally, varying lateralization of ERP responses for Turkish and German faces suggests that the underlying neural generators differ, potentially reflecting different emotional reactions to these targets. Behavioral responses showed an effect of violated expectations: German-accented Turkish-looking targets were evaluated as most competent of all targets. We suggest that bringing together neural and behavioral measures of expectancy violations, and using both visual and auditory information, yields a more complete picture of the processes underlying impression formation.

Hollywood movies provide continuous audiovisual information. Yet, information conveyed by movies address different sensory systems. For a broad variety of media applications (such as multimedia learning environments) it is important to understand the underlying cognitive principles. This project addresses the interplay of auditory and visual information during movie perception. Because auditory information is known to change basic visual processes, it is possible that movie perception and comprehension depends on stimulus modality. In this project, we report three experiments that studied how humans perceive and remember changes in visual and audiovisual movie clips. We observed basic processes of event perception (event segmentation, change detection, and memory) to be independent of stimulus modality. We thus conclude that event boundary perception is a general perceptual-cognitive mechanism and discuss these findings with respect to current cognitive psychological and media psychological theories.

The study examined individual contributions of visual and auditory information on multisensory integration across the life span. In the experiment, children, young adults, and older adults participated in a variant of the Sound-Induced Flash Illusion where participants had to either ignore beeps and report how many flashes they saw or ignore flashes and report how many beeps they heard. Collapsed across age, auditory input had a stronger effect on visual processing than vice versa. However, relative contributions of auditory and visual information interacted with age, with young adults showing evidence of auditory dominance (only auditory input affected visual processing), whereas, multisensory integration effects were more symmetrical in children and older adults. The findings have implications for many tasks that require the processing of multisensory information. (PsycINFO Database Record

In experiment 2, we applied a 2x2-design in which both, the visual and the auditory part of a simultaneous and directionally congruent motion, could independent of one another either be fast or slow. By this, we could compare audio-visual velocity signals which vary in the strength of the auditory as well as the visual motion component. If auditory information is integrated throughout all conditions, we’d expect subjects to require less contrast to perform pursuit. This should be reflected in lower PSE values in the fast auditory conditions (main effect). If, however, integration depends on the coherence of both velocities, we’d expect an interaction in the sense that a pursuit benefit only appears if both components are either fast or slow (interaction). Additionally, we want to test if subjects first have to learn the integration of auditory and visual information. We thus split up the experiment in three consecutive sessions. If integration requires learning, the benefit should become more prominent during the later sessions.