Synchronous and asynchronous presentations of auditory and visual signals: Implications for control console design

We examined inhibitory control in an oculomotor countermanding task, where the primary task required a saccadic eye movement be made to a target and a less-frequent secondary task required that the movement be halted. Previous studies have used a visual stimulus presented centrally on the fovea as the signal to stop or countermand a saccade. In these previous studies, there are at least two possible sources of saccadic inhibition: (1) sensory stimulation at the fovea can elicit a bottom-up mechanism, where a visual transient signal can delay or inhibit the developing saccade command; and (2) information based on the task instruction can be used to initiate a top-down mechanism to halt the movement. In the present study, we used both visual and auditory stop signals to test the hypothesis that the bottom-up mechanism is activated only after presentation of a foveal visual stop signal. Subjects were instructed first to look at a central spot and then to look to an eccentric visual target that appeared randomly to the left or right of center. On about one-third of the trials, a stop signal was presented. Three types of stop signals were used with equal probability: a broad-band noise burst (auditory), a central fixation spot (visual), and a combination of the auditory and visual stimuli (combined). Saccadic reaction time and stop-signal accuracy were used to calculate stop signal reaction time (SSRT), an estimate of the time required to inhibit the eye movement. Mean SSRT was longer for the auditory stop signals (201 ms) than for the signals with a foveal visual component (visual 113 ms; combined 91 ms). We conclude that a foveal visual stop signal in an oculomotor countermanding task changes the measure of inhibitory control to reflect not only inhibitory processes but also the sensory information afforded by stimulation at the fovea.

The present study compared sensitivity for auditory and visual signals in a simple detection task and in a related task in which ,9 was also imaging mental pictures and sounds. Sensitivity (rf') was reduced during imagery; within the imaging conditions, it was smaller when image and signal were both auditory or both visual than for cross-modal conditions and smaller with unfamiliar than familiar images. Likelihood ratio (Lx) was also smaller in the isomodal imaging conditions, as there were more visual false alarms during visual imagery and more auditory false alarms during auditory imagery. The data are not consistent with the assumption that d' is lower during imagery due to distraction; they do not entirely fit a channel competition model, but suggest that imagery functions as an internal signal which is confused with the external signal. Perky's (1910) effect has been difficult to explain: she found that if Os were asked to describe their images of common objects while dim facsimiles of the objects were presented before them, they reported only an imagery, not a perceptual, experience. This finding seemed paradoxical: in ordinary situations, imagery can be distinguished from real stimuli virtually 100% of the time; yet Perky's Os confused external stimuli with the images they were describing and seemed unable to discriminate the real physical signals. It is possible to explain the seeming inconsistency between Perky's experiment and everyday experience by inferring that the two events are at antipodal points on a continuum. The continuum would represent a class of conscious events characterized by activity in the sensory pathways and some central expectancies and memories, encompassing both

Two Ss identified pairs of 15 msec. one-bit auditory and visual stimuli when the interval between the stimuli varied from 0 to 500 msec. The auditory judgments were better when the auditory and visual signals were separated by 500 msec. than when the two signals were presented simultaneously, and were also better when the auditory stimulus followed the visual stimulus than when the visual stimulus was delayed.

Fidell (1982)has raised questions about the validity of our conclusions regarding performance in a bimodal detection task (Mulligan & Shaw, 1980). This reply explains why his critique is in error. Our paper is concerned with how subjects combine auditory and visual sensory information to decide whether either an auditory or a visual signal is present. Under conditions of continuous auditory and visual noise, subjects were presented with four types of stimuli: (1) auditory signal, (2) visual signal, (3) auditory plus visual signal, and (4) neither auditory nor visual signal. The subject's task was to decide whether at least one signal (visual or auditory) was presented. We presented and tested the predictions of two decision models. According to the independent decisions model, a subject makes a separate, covert, binary (yes-no) decision for each modality by comparing a measure of signal strength in that modality to the relevant decision criterion. The total number of covert yes decisions is then used to make an overt final decision: yes if one or more covert yes decisions have been made and no otherwise. In contrast, according to the integration model, a singledecision variable is formed by taking a weighted sum of the signal strength measures from the separate modalities. If this decision variable exceeds its criterion, the subject responds yes. The data from our experiment supported a version of the independent decisions model and rejected the integration model. In his critique, Fidell emphasizes a formulation of the integration model in which ad' value for bimodal detection is predicted from two separate d' values estimated from unimodal detection data. He shows that our data conform quite well to this prediction, and from this he infers that our conclusion favoring the independent decisions model is erroneous; indeed, he suggests that it must be based exclusivelyon aesthetic preference. We claim that this inference is faulty.' We will substantiate our position below by arguing that a prediction in terms of d' is not appropriate for discriminating between the two models, and that the predictions we use provide a better and more general test of the models. Earlier comparisons of independent-decisions integration models (cf. Green & Swets, 1973) have been restricted to a threshold (i.e., two-state) independentdecisions model with a nonzero probability of a detect state given noise (hence low threshold). In our paper, we consider an independent-decisions model in which sensory strength measures are treated as continuous random variables, and the internal strength measure of each modality is compared with its own criterion. The criteria selected for the auditory and visual modalities are independently adjustable, and can differ between unimodal and bimodal tasks. A consequence of these two properties is that the continuous independent decisions model can produce hit and false alarm rates that mimic the hit and false alarm rates of the integration model. This means that these models are not identifiable on the basis of the relation between bimodal and unimodal d' values. Thus, Fidell's observation that our data conform to the integration model predictions for d' is meaningless-because this conformity also applies to predictions from the independent decisions model, given appropriate choices of criteria. In our paper, we do not rely exclusively on the prediction of one model to discriminate the two theoretical alternatives. Instead, we derive predictions from each model in terms of the same statistic (probability of a no response), and these two predictions are pitted against each other. (See Green & Swets, 1973, and Sternberg, 1963, for a discussion.) A separate analysis (Shaw, 1980) shows that these and the other predictions we use differ enough between the models to provide reliable discrimination. The data we collected clearly reject the integration model and support the independent decisions model. Only if it can be argued that thereliable discrimination of one model from another confers a certain beauty on that model can we agree with Fidell that our choice of the independent decisions model is aesthetic. We claim that our tests of these models are more general in several ways than the test proposed by Fidell (and others; cf. Green & Swets, 1973). First, our predictionsapply to both threshold and continuousstate variants of the independent-decisions model (see Shaw, in press). Second, for none of the models we considered do the critical predictions depend upon subjects' maintaining fixed criteria from unimodal to bimodal detection tasks. Third, for none of our critical predictions is it required that attentional demands be invariant between unimodal and bimodal

Results from a series of experiments that use neural networks to process the visual speech signals of a male talker are presented. In these preliminary experiments, the results are limited to static images of vowels. It is demonstrated that these networks are able to extract speech information from the visual images and that this information can be used to improve automatic vowel recognition. The structure of speech and its corresponding acoustic and visual signals are reviewed. The specific data that was used in the experiments along with the network architectures and algorithms are described. The results of integrating the visual and auditory signals for vowel recognition in the presence of acoustic noise are presented. >

Based on the findings of the preceding paper, it is known that auditory and visual signals have been translated into common coordinates at the level of the superior colliculus (SC) and share a motor circuit involved in the generation of saccadic eye movements. It is not known, however, whether the translation of sensory signals into motor coordinates occurs prior to or within the SC. Nor is it known in what coordinates auditory signals observed in the SC are encoded. The present experiment tested two alternative hypotheses concerning the frame of reference of auditory signals found in the deeper layers of the SC. The hypothesis that auditory signals are encoded in head coordinates predicts that, with the head stationary, the response of auditory neurons will not be affected by variations in eye position but will be determined by the location of the sound source. The hypothesis that auditory responses encode the trajectory of the eye movement required to look to the target (motor error) predicts that the response of auditory cells will depend on both the position of the sound source and the position of the eyes in the orbit. Extracellular single-unit recordings were obtained from neurons in the SC while monkeys made delayed saccades to auditory or visual targets in a darkened room. The coordinates of auditory signals were studied by plotting auditory receptive fields while the animal fixated one of three targets placed 24 degrees apart along the horizontal plane. For 99 of 121 SC cells, the spatial location of the auditory receptive field was significantly altered by the position of the eyes in the orbit. In contrast, the responses of five sound-sensitive cells isolated in the inferior colliculus were not affected by variations in eye position. The possibility that systematic variations in the position of the pinnae associated with different fixation positions could account for these findings was controlled for by plotting auditory receptive fields while the pinnae were mechanically restrained. Under these conditions, the position of the eyes in the orbit still had a significant effect on the responsiveness of collicular neurons to auditory stimuli. The average magnitude of the shift of the auditory receptive field with changes in eye position (12.9 degrees) did not correspond to the magnitude of the shift in eye position (24 degrees). Alternative explanations for this finding were considered. One possibility is that, within the SC, there is a gradual transition from auditory signals in head coordinates to signals in motor error coordinates.(ABSTRACT TRUNCATED AT 400 WORDS)

Introduction:Application of biofeedback techniques in rehabilitation has turned into an exciting research area during the recent decade. Providing an appropriate visual or auditory biofeedback signal is the most critical requirement of a biofeedback technique. In this regard, changes in Surface Electromyography (SEMG) signals during wrist movement can be used to generate an indictable visual biofeedback signal for wrist movement rehabilitation via SEMG biofeedback. This paper proposes a novel methodology for selecting the most appropriate features out of wrist muscle SEMG signals.Methods:To this end, the surface EMG signals from flexor and extensor muscle groups during wrist joint movements were recorded and analyzed. Some linear and nonlinear features in frequency, time, and time-frequency domains were extracted from the recorded surface EMG signals of the flexor and extensor muscles. Experiments and analyses were performed on ten healthy subjects and four stroke patients with wrist muscle spasticity as the movement disorder subjects. Some heuristic feature selection measures were applied. The main motivation behind choosing applied heuristic feature selection measures was meeting. In the first step, the designed visual biofeedback signal should indicate a healthy wrist motion profile as its successful tracking by the patient guarantees rehabilitation. In addition, the visual biofeedback signal should be a smooth curve thus preventing the patient from discomfort while tracking it on a monitor during the biofeedback therapy.Results:In this pilot study, after using the introduced feature selection measures, quantitative and qualitative analyses of the extracted features indicated that Shannon entropy is the most appropriate feature for generating a visual biofeedback signal as a healthy wrist motion profile to improve the ability of stroke patients in controlling wrist joint motion. In addition, it was shown that when the wrist joint moves between a flexed and rest position, the flexor muscle EMG signal should be used for generating a visual biofeedback signal. However when the wrist joint moves between a rest position and an extended position, the extensor muscle EMG signal is appropriate for providing a visual biofeedback signal. It is worth noting that the achieved pilot study results should be confirmed by the future studies with larger samples.Conclusion:According to the obtained results, it can be concluded that among the analyzed features, the Shannon entropy was the most appropriate feature. It can be employed for generating a visual biofeedback signal for reduction of spasticity in patients with stroke.

Mandibular reaction time was measured in young and elderly volunteers. Each subject was instructed to open the jaw from the closed position and to close the jaw from the open position as quickly as possible after perceiving an auditory or visual signal. The reaction time of the elderly subjects to both signals was significantly slower than that of the young subjects. This was due almost entirely to prolongation of the premotor time component. There was no difference in the maximum velocity of the jaw movement between the two groups of subjects.

Segmenting information into smaller parts helps to process it, and this is also true for temporal information. The aim of the present article is to compare the benefits of using explicit counting in a temporal discrimination task under various marker-type conditions and to show the limits of this strategy. In Experiment 1, conditions with and without counting were compared for two implicit standard durations, .8 and 1.6 s, in connection with three marker-type conditions, which were intervals marked by: 1) two brief auditory signals (Auditory-Auditory); 2) two brief visual signals (Visual-Visual); and 3) one auditory signal followed by a visual signal (Auditory-Visual). At .8 s, marker-type differences are significant (best in audition, worse with a bimodal sequence), and remain present with an explicit counting strategy. At 1.6 s, explicit counting provides clear improvements of performance in all marker-type conditions and annihilates marker-related differences. Experiment 1 also suggests that standard deviation remains constant from .8 to 1.6 s in the counting condition, while Experiment 2 shows that when standard intervals are extended up to 4 s, explicit counting does not totally prevent variance from increasing as base duration becomes progressively longer. The benefits derived from using explicit counting in duration discrimination are argued to depend (1) on a reduction of variance in the memory process involved in the timing mechanism, and (2) on a change in the decisional process.

Compared with their performance with localized (on-key) visual stimuli, pigeons are notoriously poor at performing go/no-go discriminations when keypecking for food in the presence of auditory discriminative stimuli. The difference might reflect the fact that an aversive visual onkey stimulus signaling nonreward can be escaped by looking away and not pecking, which contributes to the measure of good discriminative performance, while an auditory stimulus cannot be escaped. In Experiment 1, discriminative performance was significantly improved by providing pigeons with a response incompatible with keypecking by which they could escape a tone S+ and a tone S−. However, the pattern, frequency, and duration of escape responses were found to be insufficient to explain the improvement. In Experiment 2, it was found that the capacity to escape only S+ or only S− enhanced discriminative performance as much as the capacity to escape both. It is theorized that the Pavlovian relationship between the absence of the discriminative stimuli and the nonoccurrence of food might transfer to the instrumental relationships learned in a go/no-go discrimination. The possibility that intermittent stimuli command more attention than continuous stimuli is also considered.

The cognitive and emotional effects of amygdala or frontal cortex lesions were compared in rats trained to time both a 50-s visual signal paired with food and an embedded 10- or 20-s auditory signal that was paired with either appetitive (food) or aversive (footshock) outcomes. When both auditory and visual signals were paired with food, control and amygdalar-lesioned rats were able to divide attention and to time both signals simultaneously, whereas when the embedded auditory signal was paired with footshock, control rats were impaired in their ability to divide attention and were able to time only one signal at a time. In contrast, amygdalar inactivation blocked this fear-related impairment and allowed rats to time both signals simultaneously, whereas rats with frontal cortex lesions demonstrated sequential processing under all conditions. These results support the proposal that the frontal cortex exerts executive control over the allocation of attentional resources, but that under stressful conditions the amygdala is crucial for the emergence of fear-evoked increments in selective attention leading to deficits in the ability to time 2 or more signals simultaneously.

Six-month-old infants’ communication in a comparative perspective: Do maternal attention and interaction matter?

Although speech perception is usually considered an auditory process, studies have shown that visual information provided by the movements of a talker’s mouth and face strongly influences what an observer perceives even when the auditory signal is clear and unambiguous. The now classic “McGurk effect” is an excellent example: the auditory syllable /bi/ presented in synchrony with a videotape of a talker saying the syllable /gi/ is usually perceived as /di/, a syllable not presented to either modality. This phenomenon has important implications for theories of speech perception which must account for how and why auditory and visual signals are integrated during phonetic processing. The purpose of the current talk is to describe some of the research that has contributed to our understanding of how visual and auditory information are used in phonetic perception. Specifically, this paper focuses on three issues: (1) what factors influence the integration of auditory and visual information; (2) the nature of the auditory and visual information used in phonetic perception; and (3) the interaction of the two sources of information during phonetic processing. The implications of these findings for theories of speech perception are discussed.

It has long been debated whether aesthetics is something uniquely human. Our ongoing project aims to empirically address comparative aesthetics in a collaboration between psychologists and biologists. Defining aesthetic responses as approach behaviour associated with sensory pleasure and corresponding physiological states, we are exploring whether humans and non-human animals share similar mechanisms in their evaluation of sexually attractive stimuli. Specifically, we compare humans and ring doves (Streptopelia risoria), a species whose courtship behaviour and physiology have been extensively investigated in behavioural and neuroendocrinology research. Both species form pair bonds, communicate using visual and auditory signals, and can be tested in laboratory conditions with conspecifics or using video and auditory stimuli. Truly comparative work involves using as similar an experimental approach as possible for both human and animal participants. Here we report our comparative methods and experimental paradigms, and some recent results.

IntroductionMultisensory signals often interact to reduce perceptual uncertainty in the environment. However, the effects and mechanisms underlying audiovisual interactions in motion perception remain unclear. In this study, we adopted the method of constant stimuli and the equivalent noise paradigm to investigate whether and how auditory motion influences the perception of visual global motion.MethodsThe visual stimuli consisted of dots moving either up-left or up-right, with motion directions sampled from a normal distribution at five levels of standard deviation. The auditory stimuli were white noise moving either laterally (leftward or rightward; Experiment 1) or diagonally (up-left or up-right; Experiment 2), forming a coarse congruent or incongruent directional relationship with the visual motion trajectories. Stationary and no-sound conditions were also included. The auditory signals were task-irrelevant and presented in spatial proximity to, but not fully overlapping with, the visual stimuli. Participants had to discriminate the direction of the visual global motion.Results and discussionAfter accounting for or eliminating the bias induced by auditory motion at the decisional level, the thresholds of visual motion perception were found to be similar across the four auditory conditions. Further analysis using the equivalent noise model confirmed that auditory motion did not influence the detection or pooling of visual motion signals. Hence, we did not find evidence to support the notion that auditory motion modulates the sensory or perceptual processing of visual global motion, delineating a boundary condition for such crossmodal interactions.