Learning multisensory neural controllers for robot arm tracking

BackgroundPlanning and executing movements requires the integration of different sensory modalities, such as vision and proprioception. However, neurological diseases like stroke can lead to full or partial loss of proprioception, resulting in impaired movements. Recent advances focused on providing additional sensory feedback to patients to compensate for the sensory loss, proving vibrotactile stimulation to be a viable option as it is inexpensive and easy to implement. Here, we test how such vibrotactile information can be integrated with visual signals to estimate the spatial location of a reach target.MethodsWe used a center-out reach paradigm with 31 healthy human participants to investigate how artificial vibrotactile stimulation can be integrated with visual-spatial cues indicating target location. Specifically, we provided multisite vibrotactile stimulation to the moving dominant arm using eccentric rotating mass (ERM) motors. As the integration of inputs across multiple sensory modalities becomes especially relevant when one of them is uncertain, we additionally modulated the reliability of visual cues. We then compared the weighing of vibrotactile and visual inputs as a function of visual uncertainty to predictions from the maximum likelihood estimation (MLE) framework to decide if participants achieve quasi-optimal integration.ResultsOur results show that participants could estimate target locations based on vibrotactile instructions. After short training, combined visual and vibrotactile cues led to higher hit rates and reduced reach errors when visual cues were uncertain. Additionally, we observed lower reaction times in trials with low visual uncertainty when vibrotactile stimulation was present. Using MLE predictions, we found that integration of vibrotactile and visual cues followed optimal integration when vibrotactile cues required the detection of one or two active motors. However, if estimating the location of a target required discriminating the intensities of two cues, integration violated MLE predictions.ConclusionWe conclude that participants can quickly learn to integrate visual and artificial vibrotactile information. Therefore, using additional vibrotactile stimulation may serve as a promising way to improve rehabilitation or the control of prosthetic devices by patients suffering loss of proprioception.

It is well established that vision plays a role in segmental speech perception, but the role of vision in prosodic speech perception is less clear. We report on the difficulties in prosodic speech perception encountered by KB after a right hemisphere stroke. In addition to musical deficits, KB was suspected of having impaired auditory prosody perception. As expected, KB was impaired on two prosody perception tasks in an auditory-only condition. We also examined whether the addition of visual prosody cues would facilitate his performance on these tasks. Unexpectedly, KB was also impaired on both tasks under visual-only and audio-visual conditions. Thus, there was no evidence that KB could integrate auditory and visual prosody information or that he could use visual cues to compensate for his deficit in the auditory domain. In contrast, KB was able to identify segmental speech information using visual cues and to use these visual cues to improve his performance when auditory segmental cues were impoverished. KB was also able to integrate audio-visual segmental information in the McGurk effect. Thus, KB’s visual deficit was specific to prosodic speech perception and, to our knowledge, this is the first reported case of such a deficit.

Future vertical lift (FVL) missions will be characterized by increased agility, degraded visual environments (DVE) and optionally piloted vehicles (OPVs). Increased agility will induce more frequent variations of linear and angular accelerations, while DVE will reduce the structure and quality of the out-the-window (OTW) scene. As rotorcrafts become faster and more agile, pilots are expected to navigate at low altitudes while traveling at high speeds. In nap of the earth (NOE) flights, the perception of self-position and orientation provided by visual, vestibular, and proprioceptive cues can vary from moment to moment due to visibility conditions and body alignment as a response to gravitoinertial forces and internally/externally induced perturbations. As a result, erroneous perceptions of the self and the environment can arise, leading ultimately to spatial disorientation (SD). In OPVs conditions, the use of different autopilot modes implies a modification of pilot role from active pilot to systems supervisor. This shift in paradigm, where pilotage is not the primary task, and where feedback from the controls is not available, has consequences. Indeed, space perception and its geometric properties can be strongly modulated by the active or passive nature of the displacement in space. In view of the link between the level of automation and Sense of Agency (SoA), it is of particular interest to examine whether agency mechanisms can modulate the level of visuo-vestibular integration in tasks of action perception and control. An experiment was conducted using the NASA Ames vertical motion simulator (VMS) to evaluate the effects of optical and gravitoinertial cues in the assessment of altitude in contour terrain flight. Seven U.S. Army pilots participated in the experiment. The aim of the proposed research was a) to establish the relative contribution of visual and gravitoinertial cues as a function of the quality of the visual cues (good vs. degraded), and the presence or absence of gravitoinertial cues, b) to determine the role of manual control vs. supervisory monitoring control on the estimation of altitude, and c) study the interactions between the nature and the quality of the sensory cues and the type of control. For the supervisory control condition, the results showed that the gravitoinertial component played a significant role in the estimation of ground height, but only in the case where the optical structure did not efficiently specify the actor-environment interaction. The improvement of the tracking performance in the visuo-vestibular setting as compared to a visual only setting when the visual cues were poor indicates some level of multisensory integration. Preliminary results for the manual control condition suggest the gravitoinertial cues contribute to an increased safety margin in presence of obstacles, particularly in DVE. Altitude accuracy and precision will then be compared for the manual and supervisory control tasks and discussed in the context of the Sense of Agency theory.

Theory predicts that animals should adjust their escape responses to the perceived predation risk. The information animals obtain about potential predation risk may differ qualitatively depending on the sensory modality by which a cue is perceived. For instance, olfactory cues may reveal better information about the presence or absence of threats, whereas visual information can reliably transmit the position and potential attack distance of a predator. While this suggests a differential use of information perceived through the two sensory channels, the relative importance of visual vs. olfactory cues when distinguishing between different predation threats is still poorly understood. Therefore, we exposed individuals of the cooperatively breeding cichlid Neolamprologus pulcher to a standardized threat stimulus combined with either predator or non-predator cues presented either visually or chemically. We predicted that flight responses towards a threat stimulus are more pronounced if cues of dangerous rather than harmless heterospecifics are presented and that N. pulcher, being an aquatic species, relies more on olfaction when discriminating between dangerous and harmless heterospecifics. N. pulcher responded faster to the threat stimulus, reached a refuge faster and entered a refuge more likely when predator cues were perceived. Unexpectedly, the sensory modality used to perceive the cues did not affect the escape response or the duration of the recovery phase. This suggests that N. pulcher are able to discriminate heterospecific cues with similar acuity when using vision or olfaction. We discuss that this ability may be advantageous in aquatic environments where the visibility conditions strongly vary over time.Significance statementThe ability to rapidly discriminate between dangerous predators and harmless heterospecifics is crucial for the survival of prey animals. In seasonally fluctuating environment, sensory conditions may change over the year and may make the use of multiple sensory modalities for heterospecific discrimination highly beneficial. Here we compared the efficacy of visual and olfactory senses in the discrimination ability of the cooperatively breeding cichlid Neolamprologus pulcher. We presented individual fish with visual or olfactory cues of predators or harmless heterospecifics and recorded their flight response. When exposed to predator cues, individuals responded faster, reached a refuge faster and were more likely to enter the refuge. Unexpectedly, the olfactory and visual senses seemed to be equally efficient in this discrimination task, suggesting that seasonal variation of water conditions experienced by N. pulcher may necessitate the use of multiple sensory channels for the same task.

Benefits of synchronous presentation of multisensory compared to unisensory cues are well established. However, the generality of such findings to children's learning with visual and haptic sensory cue pairings is unclear. Children aged 6 to 10 years (N = 180) participated in a novel tabletop category‐learning paradigm with visual, haptic, or visuohaptic informative cues. The results indicated that combinations of complimentary visual and haptic cues facilitated learning above unisensory visual cues only in 8‐year‐old children. Primarily, however, haptic information was found to dominate children's category learning across ages, particularly in the youngest children (6‐year‐olds), even with equal discriminability of haptic and visual exemplars. These findings suggest developmental changes in the ability to effectively combine unrelated visual and haptic information for categorical learning. Implications for the use of nonpertinent visuohaptic cues in learning tasks within educational settings at different ages, and in particular the dominance of haptic stimuli for children's learning, are discussed.Highlights A novel category learning task examined the role of unisensory and multisensory visual and haptic information in children's learning. Haptic cues dominated learning from 6 to 10 years of age, particularly when combined with visual information in older children. Findings infer protracted development of benefits in combining visuohaptic cues, and dominance of haptic over visual information for category learning.

We demonstrate that adaptation to novel dynamics is stronger when additional online visual cues that are congruent with the dynamics are presented during adaptation, compared with either a constant or incongruent visual cue. This effect was found regardless of whether a bidirectional or unidirectional velocity-dependent force field was presented to the participants. We propose that this effect might arise through the inclusion of this additional visual cue information within the state estimation process.

Foraging strategies of birds can influence trophic plant–insect networks with impacts on primary plant production. Recent experiments show that some forest insectivorous birds can use herbivore‐induced plant volatiles (HIPVs) to locate herbivore‐infested trees, but it is unclear how birds combine or prioritize visual and olfactory information when making foraging decisions. Here, we investigated attraction of ground‐foraging birds to HIPVs and visible prey in short vegetation on farmland in a series of foraging choice experiments. Birds showed an initial preference for HIPVs when visual information was the same for all choice options (i.e., one experimental setup had all options with visible prey, another setup with hidden prey). However, if the alternatives within an experimental setup included visible prey (without HIPV) in competition with HIPV‐only, then birds preferred the visual option over HIPVs. Our results show that olfactory cues can play an important role in birds’ foraging choices when visual information contains little variation; however, visual cues are preferred when variation is present. This suggests certain aspects of bird foraging decisions in agricultural habitats are mediated by olfactory interaction mechanisms between birds and plants. We also found that birds from variety of dietary food guilds were attracted to HIPVs; hence, the ability of birds to use plant cues is probably more general than previously thought, and may influence the biological pest control potential of birds on farmland.

Surface texture can be perceived not only from tactile, but also from auditory and visual sensory cues. In our previous psychophysical study, we demonstrated that humans can recognize surface stickiness using only one kind of sensory modality without any difficulty. However, the brain regions that would be activated by non-corresponding sensory cues, for example, auditory and visual cues, remain unknown. In this human functional MRI study, we explored brain regions associated with surface stickiness perception in each of three different sensory modalities, and sought for common neural activities across modalities. In the tactile condition, participants actually touched a sticky surface with their right index finger. In the auditory and visual conditions, audio and video clips of tactile explorations of a sticky surface were presented and participants were asked to recall the perceived stickiness as vividly as possible. Our results, based on a general linear model analysis, showed that somatosensory cortices including postcentral gyrus, anterior insula, and anterior intraparietal sulcus were significantly activated across all modalities. Moreover, we observed significant activation of primary sensory regions of each modality. A follow-up conjunction analysis identified that postcentral gyrus, anterior intraparietal sulcus, precentral gyrus, and supplementary motor area were activated in common. These findings could deepen our understanding of the surface stickiness perception in the human brain.

In the human brain, the incoming light to the retina is transformed into meaningful representations that allow us to interact with the world. In a similar vein, the RGB pixel values are transformed by a deep neural network (DNN) into meaningful representations relevant to solving a computer vision task it was trained for. Therefore, in my research, I aim to reveal insights into the visual representations in the human visual cortex and DNNs solving vision tasks. In the previous decade, DNNs have emerged as the state-of-the-art models for predicting neural responses in the human and monkey visual cortex. Research has shown that training on a task related to a brain region’s function leads to better predictivity than a randomly initialized network. Based on this observation, we proposed that we can use DNNs trained on different computer vision tasks to identify functional mapping of the human visual cortex. To validate our proposed idea, we first investigate a brain region occipital place area (OPA) using DNNs trained on scene parsing task and scene classification task. From the previous investigations about OPA’s functions, we knew that it encodes navigational affordances that require spatial information about the scene. Therefore, we hypothesized that OPA’s representation should be closer to a scene parsing model than a scene classification model as the scene parsing task explicitly requires spatial information about the scene. Our results showed that scene parsing models had representation closer to OPA than scene classification models thus validating our approach. We then selected multiple DNNs performing a wide range of computer vision tasks ranging from low-level tasks such as edge detection, 3D tasks such as surface normals, and semantic tasks such as semantic segmentation. We compared the representations of these DNNs with all the regions in the visual cortex, thus revealing the functional representations of different regions of the visual cortex. Our results highly converged with previous investigations of these brain regions validating the feasibility of the proposed approach in finding functional representations of the human brain. Our results also provided new insights into underinvestigated brain regions that can serve as starting hypotheses and promote further investigation into those brain regions. We applied the same approach to find representational insights about the DNNs. A DNN usually consists of multiple layers with each layer performing a computation leading to the final layer that performs prediction for a given task. Training on different tasks could lead to very different representations. Therefore, we first investigate at which stage does the representation in DNNs trained on different tasks starts to differ. We further investigate if the DNNs trained on similar tasks lead to similar representations and on dissimilar tasks lead to more dissimilar representations. We selected the same set of DNNs used in the previous work that were trained on the Taskonomy dataset on a diverse range of 2D, 3D and semantic tasks. Then, given a DNN trained on a particular task, we compared the representation of multiple layers to corresponding layers in other DNNs. From this analysis, we aimed to reveal where in the network architecture task-specific representation is prominent. We found that task specificity increases as we go deeper into the DNN architecture and similar tasks start to cluster in groups. We found that the grouping we found using representational similarity was highly correlated with grouping based on transfer learning thus creating an interesting application of the approach to model selection in transfer learning. During previous works, several new measures were introduced to compare DNN representations. So, we identified the commonalities in different measures and unified different measures into a single framework referred to as duality diagram similarity. This work opens up new possibilities for similarity measures to understand DNN representations. While demonstrating a much higher correlation with transfer learning than previous state-of-the-art measures we extend it to understanding layer-wise representations of models trained on the Imagenet and Places dataset using different tasks and demonstrate its applicability to layer selection for transfer learning. In all the previous works, we used the task-specific DNN representations to understand the representations in the human visual cortex and other DNNs. We were able to interpret our findings in terms of computer vision tasks such as edge detection, semantic segmentation, depth estimation, etc. however we were not able to map the representations to human interpretable concepts. Therefore in our most recent work, we developed a new method that associates individual artificial neurons with human interpretable concepts. Overall, the works in this thesis revealed new insights into the representation of the visual cortex and DNNs...

The relationship between the size and structure of a species' brain and its cognitive capacity has long interested scientists. Generally, this work relates interspecific variation in brain anatomy with performance on a variety of cognitive tasks. However, brains are known to show considerable short-term plasticity in response to a range of social, ecological and environmental factors. Despite this, we have a remarkably poor understanding of how this impacts on an animal's cognitive performance. Here, we non-invasively manipulated the relative size of brain regions associated with processing visual and chemical information in fish (the optic tectum and olfactory bulbs, respectively). We then tested performance in a cognitive task in which information from the two sensory modalities was in conflict. Although the fish could effectively use both visual and chemical information if presented in isolation, when they received cues from both modalities simultaneously, those with a relatively better developed optic tectum showed a greater reliance on visual information, while individuals with relatively better developed olfactory bulbs showed a greater reliance on chemical information. These results suggest that short-term changes in brain structure, possibly resulting from an attempt to minimize the costs of developing unnecessary but energetically expensive brain regions, may have marked effects on cognitive performance.This article is part of the theme issue 'Causes and consequences of individual differences in cognitive abilities'.

How the CNS transforms visual information of object properties into motor commands for manipulation is not well understood. We designed novel apparatus and protocols in which human subjects had to learn manipulations in two different contexts. The first task involved manipulating a U-shaped object that can afford two actions by grasping different parts of the same object. The second task involved manipulating two L-shaped objects that were posed at different orientations. In both experiments, subjects learned the manipulation over consecutive trials in one context before switching to a different context. For both objects and tasks, the visual geometric cues were effective in eliciting anticipatory control with little error at the beginning of learning of the first context. However, subjects failed to use the visual information to the same extent when switching to the second context as sensorimotor memory built through eight consecutive repetitions in the first context exerted a strong interference on subjects' ability to use visual cues again when the context changed. A follow-up experiment where subjects were exposed to a pseudorandom sequence of context switches with the U-shaped object revealed that the interference caused by the preceding context persisted even when subjects switched context after only one trial. Our results suggest that learning generalization of dexterous manipulation is fundamentally limited by context-specific learning of motor actions and competition between vision-based motor planning and sensorimotor memory.

We design a personalized human-robot environment for social learning for individuals with autism spectrum disorders (ASD). In order to define an individual's profile, we posit that the individual's reliance on proprioceptive and kinematic visual cues should affect the way the individual suffering from ASD interacts with a social agent (human/robot/virtual agent). In this paper, we assess the potential link between recognition performances of body/facial expressions of emotion of increasing complexity, emotion recognition on platforms with different visual features (two mini-humanoid robots, a virtual agent, and a human), and proprioceptive and visual cues integration of an individual. First, we describe the design of the EMBODI-EMO database containing videos of controlled body/facial expressions of emotions from various platforms. We explain how we validated this database with typically developed (TD) individuals. Then, we investigate the relationship between emotion recognition and proprioceptive and visual profiles of TD individuals and individuals with ASD. For TD individuals, our results indicate a relationship between profiles and emotion recognition. As expected, we show that TD individuals that rely more heavily on visual cues yield better recognition scores. However, we found that TD individuals relying on proprioception have better recognition scores, going against our hypothesis. Finally, participants with ASD relying more heavily on proprioceptive cues have lower emotion recognition scores on all conditions than participants relying on visual cues.

Head direction (HD) neurons, signalling facing direction, generate a signal that is primarily anchored to the outside world by visual inputs. We investigated the route for visual landmark information into the HD system in rats. There are two candidates: an evolutionarily older, larger subcortical retino-tectal pathway and a more recently evolved, smaller cortical retino-geniculo-striate pathway. We disrupted the cortical pathway by lesioning the dorsal lateral geniculate thalamic nuclei bilaterally, and recorded HD cells in the postsubicular cortex as rats foraged in a visual-cue-controlled enclosure. In lesioned rats we found the expected number of postsubicular HD cells. Although directional tuning curves were broader across a trial, this was attributable to the increased instability of otherwise normal-width tuning curves. Tuning curves were also poorly responsive to polarizing visual landmarks and did not distinguish cues based on their visual pattern. Thus, the retino-geniculo-striate pathway is not crucial for the generation of an underlying, tightly tuned directional signal but does provide the main route for vision-based anchoring of the signal to the outside world, even when visual cues are high in contrast and low in detail. KEY POINTS: Head direction (HD) cells indicate the facing direction of the head, using visual landmarks to distinguish directions. In rats, we investigated whether this visual information is routed through the thalamus to the visual cortex or arrives via the superior colliculus, which is a phylogenetically older and (in rodents) larger pathway. We lesioned the thalamic dorsal lateral geniculate nucleus (dLGN) in rats and recorded the responsiveness of cortical HD cells to visual cues. We found that cortical HD cells had normal tuning curves, but these were slightly more unstable during a trial. Most notably, HD cells in dLGN-lesioned animals showed little ability to distinguish highly distinct cues and none to distinguish more similar cues. These results suggest that directional processing of visual landmarks in mammals requires the geniculo-cortical pathway, which raises questions about when and how visual directional landmark processing appeared during evolution.

Ants use the panoramic skyline as a visual cue during navigation

The VR-based circular tracking movement evaluation system (CES) was developed to quantitatively assess visuomotor control. The virtual stick, a component of the CES, provides visual cues in the virtual environment and haptic feedback when holding the controller. This study examined the effects of stick presence and presentation order on control accuracy for distance, angle, and angular velocity. Twenty-seven participants (12 females; mean age 23.3 ± 2.3 years) performed tasks in the frontal plane followed by the sagittal plane. In each plane, the stick was visible for the first 1-3 revolutions and invisible for the subsequent 4-6 revolutions in the invisible condition, with the reverse order in the visible condition. In the invisible condition, control accuracy with the stick was 1.10 times higher for distance only in the sagittal plane, and 1.13 and 1.09 times higher for angle and angular velocity in the frontal plane, and 1.11 and 1.08 times higher in the sagittal plane. No significant differences were observed in the visible condition. The improved control accuracy when the stick was visible is likely due to enhanced precision in constructing the reference frame, internal models, body coordinates, and effective multisensory integration of visual and haptic information. Such visual information may enable fine control in virtual environment-based applications, including games and surgical simulations.