Socially meaningful visual context either enhances or inhibits vocalisation processing in the macaque brain

Although speech perception is usually considered an auditory process, studies have shown that visual information provided by the movements of a talker’s mouth and face strongly influences what an observer perceives even when the auditory signal is clear and unambiguous. The now classic “McGurk effect” is an excellent example: the auditory syllable /bi/ presented in synchrony with a videotape of a talker saying the syllable /gi/ is usually perceived as /di/, a syllable not presented to either modality. This phenomenon has important implications for theories of speech perception which must account for how and why auditory and visual signals are integrated during phonetic processing. The purpose of the current talk is to describe some of the research that has contributed to our understanding of how visual and auditory information are used in phonetic perception. Specifically, this paper focuses on three issues: (1) what factors influence the integration of auditory and visual information; (2) the nature of the auditory and visual information used in phonetic perception; and (3) the interaction of the two sources of information during phonetic processing. The implications of these findings for theories of speech perception are discussed.

BackgroundPlanning and executing movements requires the integration of different sensory modalities, such as vision and proprioception. However, neurological diseases like stroke can lead to full or partial loss of proprioception, resulting in impaired movements. Recent advances focused on providing additional sensory feedback to patients to compensate for the sensory loss, proving vibrotactile stimulation to be a viable option as it is inexpensive and easy to implement. Here, we test how such vibrotactile information can be integrated with visual signals to estimate the spatial location of a reach target.MethodsWe used a center-out reach paradigm with 31 healthy human participants to investigate how artificial vibrotactile stimulation can be integrated with visual-spatial cues indicating target location. Specifically, we provided multisite vibrotactile stimulation to the moving dominant arm using eccentric rotating mass (ERM) motors. As the integration of inputs across multiple sensory modalities becomes especially relevant when one of them is uncertain, we additionally modulated the reliability of visual cues. We then compared the weighing of vibrotactile and visual inputs as a function of visual uncertainty to predictions from the maximum likelihood estimation (MLE) framework to decide if participants achieve quasi-optimal integration.ResultsOur results show that participants could estimate target locations based on vibrotactile instructions. After short training, combined visual and vibrotactile cues led to higher hit rates and reduced reach errors when visual cues were uncertain. Additionally, we observed lower reaction times in trials with low visual uncertainty when vibrotactile stimulation was present. Using MLE predictions, we found that integration of vibrotactile and visual cues followed optimal integration when vibrotactile cues required the detection of one or two active motors. However, if estimating the location of a target required discriminating the intensities of two cues, integration violated MLE predictions.ConclusionWe conclude that participants can quickly learn to integrate visual and artificial vibrotactile information. Therefore, using additional vibrotactile stimulation may serve as a promising way to improve rehabilitation or the control of prosthetic devices by patients suffering loss of proprioception.

The relationship between the size and structure of a species' brain and its cognitive capacity has long interested scientists. Generally, this work relates interspecific variation in brain anatomy with performance on a variety of cognitive tasks. However, brains are known to show considerable short-term plasticity in response to a range of social, ecological and environmental factors. Despite this, we have a remarkably poor understanding of how this impacts on an animal's cognitive performance. Here, we non-invasively manipulated the relative size of brain regions associated with processing visual and chemical information in fish (the optic tectum and olfactory bulbs, respectively). We then tested performance in a cognitive task in which information from the two sensory modalities was in conflict. Although the fish could effectively use both visual and chemical information if presented in isolation, when they received cues from both modalities simultaneously, those with a relatively better developed optic tectum showed a greater reliance on visual information, while individuals with relatively better developed olfactory bulbs showed a greater reliance on chemical information. These results suggest that short-term changes in brain structure, possibly resulting from an attempt to minimize the costs of developing unnecessary but energetically expensive brain regions, may have marked effects on cognitive performance.This article is part of the theme issue 'Causes and consequences of individual differences in cognitive abilities'.

Theory predicts that animals should adjust their escape responses to the perceived predation risk. The information animals obtain about potential predation risk may differ qualitatively depending on the sensory modality by which a cue is perceived. For instance, olfactory cues may reveal better information about the presence or absence of threats, whereas visual information can reliably transmit the position and potential attack distance of a predator. While this suggests a differential use of information perceived through the two sensory channels, the relative importance of visual vs. olfactory cues when distinguishing between different predation threats is still poorly understood. Therefore, we exposed individuals of the cooperatively breeding cichlid Neolamprologus pulcher to a standardized threat stimulus combined with either predator or non-predator cues presented either visually or chemically. We predicted that flight responses towards a threat stimulus are more pronounced if cues of dangerous rather than harmless heterospecifics are presented and that N. pulcher, being an aquatic species, relies more on olfaction when discriminating between dangerous and harmless heterospecifics. N. pulcher responded faster to the threat stimulus, reached a refuge faster and entered a refuge more likely when predator cues were perceived. Unexpectedly, the sensory modality used to perceive the cues did not affect the escape response or the duration of the recovery phase. This suggests that N. pulcher are able to discriminate heterospecific cues with similar acuity when using vision or olfaction. We discuss that this ability may be advantageous in aquatic environments where the visibility conditions strongly vary over time.Significance statementThe ability to rapidly discriminate between dangerous predators and harmless heterospecifics is crucial for the survival of prey animals. In seasonally fluctuating environment, sensory conditions may change over the year and may make the use of multiple sensory modalities for heterospecific discrimination highly beneficial. Here we compared the efficacy of visual and olfactory senses in the discrimination ability of the cooperatively breeding cichlid Neolamprologus pulcher. We presented individual fish with visual or olfactory cues of predators or harmless heterospecifics and recorded their flight response. When exposed to predator cues, individuals responded faster, reached a refuge faster and were more likely to enter the refuge. Unexpectedly, the olfactory and visual senses seemed to be equally efficient in this discrimination task, suggesting that seasonal variation of water conditions experienced by N. pulcher may necessitate the use of multiple sensory channels for the same task.

Humans learn multisensory eye-hand coordination starting from infancy without supervision. For an example, they learn to track their hands by exploiting various sensory modalities, such as vision and proprioception. This integration occurs as they learn to perceive the world around them and their relationship to it. Most prior work has focused on the role of vision, as it is a primary sensory source for humans. However, it is interesting to study how vision and proprioception interact. We propose a system which combines visual and proprioceptive information to learn the eye-hand coordination skills that enable a robot to fixate its camera gaze on the end effector of its arm. In our model, visual cues are part of the feedback control loop, whereas proprioceptive cues are part of a feedforward control loop. Both controllers, as well as the sensory transform from raw visual information to a neural sensory representation are learned as the robot performs motor babbling movements. Visual information is encoded by sparse coding. The basis functions that emerge are similar to the receptive fields in the human visual cortex. An actor-critic reinforcement learning algorithm is used to drive eye motor neurons fusing visual and proprioceptive cues. We model and test the system in the iCub simulation environment. Our results suggest that these sensory modalities are capable of jointly learning model parameters to perform the tracking task. The evolved policy has characteristics that are qualitatively similar to the human oculomotor plant.

Decision letter: Visual and motor signatures of locomotion dynamically shape a population code for feature detection in Drosophila

Neurons in the superior colliculus (SC), a prominent midbrain structure, are able to synthesize information from different senses. This synthesis plays an important role in determining whether SC-mediated orientation behaviors will be initiated. In some circumstances, multisensory integration in the SC is evident as a response that is significantly enhanced above that evoked by the most effective single-modality stimulus. It can sometimes even exceed the arithmetic sum of the single-modality responses. In other circumstances, multisensory integration is evident as response depression, an effect sometimes powerful enough to eliminate even robust single-modality responses. The conditions that produce multisensory enhancement also increase the probability of orientation responses, and those that produce multisensory response depression decrease the probability of orientation responses. Although one might posit that the capability to integrate cross-modal cues (and, in this case, alter overt behavior) would be evident in all neurons capable of responding to stimuli from two or more sensory modalities, this turns out to be incorrect. When descending influences from the cortex are temporarily inactivated, SC neurons are rendered unable to synthesize their multiple sensory inputs, and animals no longer show enhanced orientation responses. Nevertheless, the ability to respond to cues from multiple sensory modalities is retained at both the single neuron and behavioral levels. Two cortical areas have been implicated in controlling these midbrain processes: the anterior ectosylvian sulcus and the rostral lateral suprasylvian sulcus.

An image is worth a thousand words, conveying information that goes beyond the mere visual content therein. In this paper, we study the intent behind social media images with an aim to analyze how visual information can facilitate recognition of human intent. Towards this goal, we introduce an intent dataset, Intentonomy, comprising 14K images covering a wide range of everyday scenes. These images are manually annotated with 28 intent categories derived from a social psychology taxonomy. We then systematically study whether, and to what extent, commonly used visual information, i.e., object and context, contribute to human motive understanding. Based on our findings, we conduct further study to quantify the effect of attending to object and context classes as well as textual information in the form of hashtags when training an intent classifier. Our results quantitatively and qualitatively shed light on how visual and textual information can produce observable effects when predicting intent. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>

Females of many species choose mates using multiple sensory modalities. Multimodal noise may arise, however, in dense aggregations of animals communicating via multiple sensory modalities. Some evidence suggests multimodal signals may not always improve receiver decision-making performance. When sensory systems process input from multimodal signal sources, multimodal noise may arise and potentially complicate decision-making due to the demands on cognitive integration tasks. We tested female túngara frog, Physalaemus (=Engystomops) pustulosus, responses to male mating signals in noise from multiple sensory modalities (acoustic and visual). Noise treatments were partitioned into three categories: acoustic, visual, and multimodal. We used natural calls from conspecifics and heterospecifics for acoustic noise. Robotic frogs were employed as either visual signal components (synchronous vocal sac inflation with call) or visual noise (asynchronous vocal sac inflation with call). Females expressed a preference for the typically more attractive call in the presence of unimodal noise. However, during multimodal signal and noise treatments (robofrogs employed with background noise), females failed to express a preference for the typically attractive call in the presence of conspecific chorus noise. We found that social context and temporal synchrony of multimodal signaling components are important for multimodal communication. Our results demonstrate that multimodal signals have the potential to increase the complexity of the sensory scene and reduce the efficacy of female decision making.

Seismic communication and mate choice in wolf spiders: components of male seismic signals and mating success

Influence of a sensorimotor conflict on the memorization of a path traveled in virtual reality

Fidell (1982)has raised questions about the validity of our conclusions regarding performance in a bimodal detection task (Mulligan & Shaw, 1980). This reply explains why his critique is in error. Our paper is concerned with how subjects combine auditory and visual sensory information to decide whether either an auditory or a visual signal is present. Under conditions of continuous auditory and visual noise, subjects were presented with four types of stimuli: (1) auditory signal, (2) visual signal, (3) auditory plus visual signal, and (4) neither auditory nor visual signal. The subject's task was to decide whether at least one signal (visual or auditory) was presented. We presented and tested the predictions of two decision models. According to the independent decisions model, a subject makes a separate, covert, binary (yes-no) decision for each modality by comparing a measure of signal strength in that modality to the relevant decision criterion. The total number of covert yes decisions is then used to make an overt final decision: yes if one or more covert yes decisions have been made and no otherwise. In contrast, according to the integration model, a singledecision variable is formed by taking a weighted sum of the signal strength measures from the separate modalities. If this decision variable exceeds its criterion, the subject responds yes. The data from our experiment supported a version of the independent decisions model and rejected the integration model. In his critique, Fidell emphasizes a formulation of the integration model in which ad' value for bimodal detection is predicted from two separate d' values estimated from unimodal detection data. He shows that our data conform quite well to this prediction, and from this he infers that our conclusion favoring the independent decisions model is erroneous; indeed, he suggests that it must be based exclusivelyon aesthetic preference. We claim that this inference is faulty.' We will substantiate our position below by arguing that a prediction in terms of d' is not appropriate for discriminating between the two models, and that the predictions we use provide a better and more general test of the models. Earlier comparisons of independent-decisions integration models (cf. Green & Swets, 1973) have been restricted to a threshold (i.e., two-state) independentdecisions model with a nonzero probability of a detect state given noise (hence low threshold). In our paper, we consider an independent-decisions model in which sensory strength measures are treated as continuous random variables, and the internal strength measure of each modality is compared with its own criterion. The criteria selected for the auditory and visual modalities are independently adjustable, and can differ between unimodal and bimodal tasks. A consequence of these two properties is that the continuous independent decisions model can produce hit and false alarm rates that mimic the hit and false alarm rates of the integration model. This means that these models are not identifiable on the basis of the relation between bimodal and unimodal d' values. Thus, Fidell's observation that our data conform to the integration model predictions for d' is meaningless-because this conformity also applies to predictions from the independent decisions model, given appropriate choices of criteria. In our paper, we do not rely exclusively on the prediction of one model to discriminate the two theoretical alternatives. Instead, we derive predictions from each model in terms of the same statistic (probability of a no response), and these two predictions are pitted against each other. (See Green & Swets, 1973, and Sternberg, 1963, for a discussion.) A separate analysis (Shaw, 1980) shows that these and the other predictions we use differ enough between the models to provide reliable discrimination. The data we collected clearly reject the integration model and support the independent decisions model. Only if it can be argued that thereliable discrimination of one model from another confers a certain beauty on that model can we agree with Fidell that our choice of the independent decisions model is aesthetic. We claim that our tests of these models are more general in several ways than the test proposed by Fidell (and others; cf. Green & Swets, 1973). First, our predictionsapply to both threshold and continuousstate variants of the independent-decisions model (see Shaw, in press). Second, for none of the models we considered do the critical predictions depend upon subjects' maintaining fixed criteria from unimodal to bimodal detection tasks. Third, for none of our critical predictions is it required that attentional demands be invariant between unimodal and bimodal

Humans constantly process and integrate sensory input from multiple sensory modalities. However, the amount of input that can be processed is constrained by limited attentional resources. A matter of ongoing debate is whether attentional resources are shared across sensory modalities, and whether multisensory integration is dependent on attentional resources. Previous research suggested that the distribution of attentional resources across sensory modalities depends on the the type of tasks. Here, we tested a novel task combination in a dual task paradigm: Participants performed a self-terminated visual search task and a localization task in either separate sensory modalities (i.e., haptics and vision) or both within the visual modality. Tasks considerably interfered. However, participants performed the visual search task faster when the localization task was performed in the tactile modality in comparison to performing both tasks within the visual modality. This finding indicates that tasks performed in separate sensory modalities rely in part on distinct attentional resources. Nevertheless, participants integrated visuotactile information optimally in the localization task even when attentional resources were diverted to the visual search task. Overall, our findings suggest that visual search and tactile localization partly rely on distinct attentional resources, and that optimal visuotactile integration is not dependent on attentional resources.

Evidence for a specific role for muscarinic receptors in crossmodal object recognition in rats

Humans have the ability to identify external physical properties because skilled object manipulation requires knowledge, or internal models, of object dynamics relating to applied force or movement. The accuracy of identification depends on those of the input and output signal to the object : motor command from brain as the input and information from multiple sensory modalities as the output. In this paper, control strategies of isometric contraction by hand are investigated. Subjects are asked to generate constant levels of force under two conditions : force production tasks a) by self scaling without visual feedback and b) with visual feed back of reference force levels. The results show subjective assessments of force are larger than objective one, and it is indicated that the variability is composed of signal dependent noise and error of human intention. It is also demonstrated that visual feedback information has an important role of generating accurate reference force level.