Context-Dependent Learning Interferes with Visuomotor Transformations for Manipulation Planning

We demonstrate that adaptation to novel dynamics is stronger when additional online visual cues that are congruent with the dynamics are presented during adaptation, compared with either a constant or incongruent visual cue. This effect was found regardless of whether a bidirectional or unidirectional velocity-dependent force field was presented to the participants. We propose that this effect might arise through the inclusion of this additional visual cue information within the state estimation process.

Humans learn multisensory eye-hand coordination starting from infancy without supervision. For an example, they learn to track their hands by exploiting various sensory modalities, such as vision and proprioception. This integration occurs as they learn to perceive the world around them and their relationship to it. Most prior work has focused on the role of vision, as it is a primary sensory source for humans. However, it is interesting to study how vision and proprioception interact. We propose a system which combines visual and proprioceptive information to learn the eye-hand coordination skills that enable a robot to fixate its camera gaze on the end effector of its arm. In our model, visual cues are part of the feedback control loop, whereas proprioceptive cues are part of a feedforward control loop. Both controllers, as well as the sensory transform from raw visual information to a neural sensory representation are learned as the robot performs motor babbling movements. Visual information is encoded by sparse coding. The basis functions that emerge are similar to the receptive fields in the human visual cortex. An actor-critic reinforcement learning algorithm is used to drive eye motor neurons fusing visual and proprioceptive cues. We model and test the system in the iCub simulation environment. Our results suggest that these sensory modalities are capable of jointly learning model parameters to perform the tracking task. The evolved policy has characteristics that are qualitatively similar to the human oculomotor plant.

BackgroundPlanning and executing movements requires the integration of different sensory modalities, such as vision and proprioception. However, neurological diseases like stroke can lead to full or partial loss of proprioception, resulting in impaired movements. Recent advances focused on providing additional sensory feedback to patients to compensate for the sensory loss, proving vibrotactile stimulation to be a viable option as it is inexpensive and easy to implement. Here, we test how such vibrotactile information can be integrated with visual signals to estimate the spatial location of a reach target.MethodsWe used a center-out reach paradigm with 31 healthy human participants to investigate how artificial vibrotactile stimulation can be integrated with visual-spatial cues indicating target location. Specifically, we provided multisite vibrotactile stimulation to the moving dominant arm using eccentric rotating mass (ERM) motors. As the integration of inputs across multiple sensory modalities becomes especially relevant when one of them is uncertain, we additionally modulated the reliability of visual cues. We then compared the weighing of vibrotactile and visual inputs as a function of visual uncertainty to predictions from the maximum likelihood estimation (MLE) framework to decide if participants achieve quasi-optimal integration.ResultsOur results show that participants could estimate target locations based on vibrotactile instructions. After short training, combined visual and vibrotactile cues led to higher hit rates and reduced reach errors when visual cues were uncertain. Additionally, we observed lower reaction times in trials with low visual uncertainty when vibrotactile stimulation was present. Using MLE predictions, we found that integration of vibrotactile and visual cues followed optimal integration when vibrotactile cues required the detection of one or two active motors. However, if estimating the location of a target required discriminating the intensities of two cues, integration violated MLE predictions.ConclusionWe conclude that participants can quickly learn to integrate visual and artificial vibrotactile information. Therefore, using additional vibrotactile stimulation may serve as a promising way to improve rehabilitation or the control of prosthetic devices by patients suffering loss of proprioception.

Decades of research on visual information for speech has demonstrated the informativeness of visual cues to enhance (Cho et al., 2020; Kawase et al., 2014) and influence (MacDonald and McGurk, 1978) speech perception. However, it is unclear which specific visual cues are spatially and temporally correlated to certain features of speech segments. This study explored visual cues of voicing during the production of Canadian English stops using facial recognition technology (Baltrušaitis et al., 2018) and manual coding (using ELAN 2022). We recorded the audio along with two videos, capturing both front and side views simultaneously, from six native Canadian English speakers. We paid special attention to the throat (larynx), chin, and neck areas which have been understudied by the previous literature. Preliminary data shows expanding movement in the submental triangle and throat during the production of voiced stops compared to voiceless stops. This finding supports tongue body lowering and larynx lowering found in the production of English voiced stops (Westbury, 1983). The comparison between utterance initial stops with post-vocalic stops shows that certain visual cues may be related to phonological voicing categorization irrespective of actual voicing during closure, while others reflect phonetic voicing reality.

Benefits of synchronous presentation of multisensory compared to unisensory cues are well established. However, the generality of such findings to children's learning with visual and haptic sensory cue pairings is unclear. Children aged 6 to 10 years (N = 180) participated in a novel tabletop category‐learning paradigm with visual, haptic, or visuohaptic informative cues. The results indicated that combinations of complimentary visual and haptic cues facilitated learning above unisensory visual cues only in 8‐year‐old children. Primarily, however, haptic information was found to dominate children's category learning across ages, particularly in the youngest children (6‐year‐olds), even with equal discriminability of haptic and visual exemplars. These findings suggest developmental changes in the ability to effectively combine unrelated visual and haptic information for categorical learning. Implications for the use of nonpertinent visuohaptic cues in learning tasks within educational settings at different ages, and in particular the dominance of haptic stimuli for children's learning, are discussed.Highlights A novel category learning task examined the role of unisensory and multisensory visual and haptic information in children's learning. Haptic cues dominated learning from 6 to 10 years of age, particularly when combined with visual information in older children. Findings infer protracted development of benefits in combining visuohaptic cues, and dominance of haptic over visual information for category learning.

Grasping behaviour involves the integration of current and historical knowledge about an object, a process that can be influenced by sensory uncertainty. In the present study, participants simultaneously interacted with a visual cue and a haptic cue before reaching to grasp a target object. The visual cue was either congruent (equal in size to haptic cue and target) or incongruent (larger than haptic cue and target). To enhance sensory uncertainty, we manipulated the proportion of congruent trials to be either 80 or 20%. We compared grasp kinematics and forces between congruent and incongruent trials and between the 20 and 80% proportion congruency groups. We also studied the effects of trial history by comparing the performance of congruent and incongruent trials preceded by either the same or opposite trial type. Proportion congruency did not affect temporal kinematics but did affect maximum grip aperture (MGA) as the 80% proportion congruency group used a greater MGA, regardless of trial type. For grasping forces, an interaction effect showed that the 20% proportion congruency group used a greater peak load force on congruent trials. Incongruent trials that followed congruent trials had decreased movement time, increased MGA and increased grasping forces, relative to those that followed incongruent trials. We interpret the data to suggest that the grasp control system integrates multisensory information using flexible, yet specific criteria regarding task constraints. The prevention of collision error (i.e., an inadequate MGA when contacting the target) may be one guiding principle in the control process.

Foraging strategies of birds can influence trophic plant–insect networks with impacts on primary plant production. Recent experiments show that some forest insectivorous birds can use herbivore‐induced plant volatiles (HIPVs) to locate herbivore‐infested trees, but it is unclear how birds combine or prioritize visual and olfactory information when making foraging decisions. Here, we investigated attraction of ground‐foraging birds to HIPVs and visible prey in short vegetation on farmland in a series of foraging choice experiments. Birds showed an initial preference for HIPVs when visual information was the same for all choice options (i.e., one experimental setup had all options with visible prey, another setup with hidden prey). However, if the alternatives within an experimental setup included visible prey (without HIPV) in competition with HIPV‐only, then birds preferred the visual option over HIPVs. Our results show that olfactory cues can play an important role in birds’ foraging choices when visual information contains little variation; however, visual cues are preferred when variation is present. This suggests certain aspects of bird foraging decisions in agricultural habitats are mediated by olfactory interaction mechanisms between birds and plants. We also found that birds from variety of dietary food guilds were attracted to HIPVs; hence, the ability of birds to use plant cues is probably more general than previously thought, and may influence the biological pest control potential of birds on farmland.

It is well established that vision plays a role in segmental speech perception, but the role of vision in prosodic speech perception is less clear. We report on the difficulties in prosodic speech perception encountered by KB after a right hemisphere stroke. In addition to musical deficits, KB was suspected of having impaired auditory prosody perception. As expected, KB was impaired on two prosody perception tasks in an auditory-only condition. We also examined whether the addition of visual prosody cues would facilitate his performance on these tasks. Unexpectedly, KB was also impaired on both tasks under visual-only and audio-visual conditions. Thus, there was no evidence that KB could integrate auditory and visual prosody information or that he could use visual cues to compensate for his deficit in the auditory domain. In contrast, KB was able to identify segmental speech information using visual cues and to use these visual cues to improve his performance when auditory segmental cues were impoverished. KB was also able to integrate audio-visual segmental information in the McGurk effect. Thus, KB’s visual deficit was specific to prosodic speech perception and, to our knowledge, this is the first reported case of such a deficit.

An object can be used in multiple contexts, each requiring different hand actions. How the central nervous system builds and maintains memory of such dexterous manipulations remains unclear. We conducted experiments in which human subjects had to learn and recall manipulations performed in two contexts, A and B. Both contexts involved lifting the same L-shaped object whose geometry cued its asymmetrical mass distribution. Correct performance required producing a torque on the vertical handle at object lift onset to prevent it from tilting. The torque direction depended on the context, i.e., object orientation, which was changed by 180° object rotation about a vertical axis. With an A1B1A2 context switching paradigm, subjects learned A1 in the first block of eight trials as indicated by a torque approaching the required one. However, subjects made large errors in anticipating the required torque when switching to B1 immediately after A1 (negative transfer), as well as when they had to recall A1 when switching to A2 after learning B through another block of eight lifts (retrieval interference). Classic sensorimotor learning theories attribute such interferences to multi-rate, multi-state error-driven updates of internal models. However, by systematically changing the interblock break duration and within-block number of trials, our results suggest an alternative explanation underlying interference and retention of dexterous manipulation. Specifically, we identified and quantified through a novel computational model the nonlinear interaction between two sensorimotor mechanisms: a short-lived, context-independent, use-dependent sensorimotor memory and a context-sensitive, error-based learning process.

Nonverbal and other visual cues are well established as a critical component of human communication. Under most circumstances, visual information is available to aid in the comprehension and interpretation of spoken language. Citing these facts, many L2 assessment researchers have studied video-mediated listening tests through score comparisons with audio tests, by measuring the amount of time spent watching, and by attempting to determine examinee viewing behavior through self-reports. However, the specific visual cues to which examinees attend have heretofore not been measured objectively. The present research employs eye-tracking methodology to determine the amounts of time 12 participants viewed specific visual cues on a six-item, video-mediated L2 listening test. Seventy-two scanpath-overlaid videos of viewing behavior were manually coded for visual cues at 0.10-second intervals. Cued retrospective interviews based on eye-tracking data provided reasons for the observed behaviors. Faces were found to occupy the majority (81.74%) of visual dwell time, with participants largely splitting their time between the speaker’s eyes and mouth. Detected gesture viewing was negligible. The reason given for most viewing behavior was determining characters’ emotional states. These findings suggest that the primary difference between audio- and video-mediated L2 listening tests of conversational content is the absence or presence of facial expressions.

Virtual audio systems are commonly used today to create auditory environments for various listening tasks. The dimensions and absorption characteristics of the surrounding real visual space often do not match the virtual auditory listening space. This mismatch in spatial information has been shown to have a small or negligible effect in the perceived direction of sound sources. However, spatial cues can have a large effect on perceived auditory distance. The most veridical auditory spatial renderings are ones that match the visual space around the observer. Several experiments were conducted to investigate the effects of visual spatial information and head motion cues on perceived auditory distance. In the first experimental condition, naive observers were deprived of all visual cues while making judgments on the egocentric distance of sound sources played over loudspeakers. In the second condition, subjects were presented with mismatched visual and auditory spatial cues via a manikin. The acoustic manikin was either non‐moving or would rotate in correlation with the listener’s head motion. In the last (control) condition, sound sources were presented from loudspeakers with matching visual and auditory cues. The effects of the visual spatial cues and head motion cues on auditory distance judgments will be described.

Head direction (HD) neurons, signalling facing direction, generate a signal that is primarily anchored to the outside world by visual inputs. We investigated the route for visual landmark information into the HD system in rats. There are two candidates: an evolutionarily older, larger subcortical retino-tectal pathway and a more recently evolved, smaller cortical retino-geniculo-striate pathway. We disrupted the cortical pathway by lesioning the dorsal lateral geniculate thalamic nuclei bilaterally, and recorded HD cells in the postsubicular cortex as rats foraged in a visual-cue-controlled enclosure. In lesioned rats we found the expected number of postsubicular HD cells. Although directional tuning curves were broader across a trial, this was attributable to the increased instability of otherwise normal-width tuning curves. Tuning curves were also poorly responsive to polarizing visual landmarks and did not distinguish cues based on their visual pattern. Thus, the retino-geniculo-striate pathway is not crucial for the generation of an underlying, tightly tuned directional signal but does provide the main route for vision-based anchoring of the signal to the outside world, even when visual cues are high in contrast and low in detail. KEY POINTS: Head direction (HD) cells indicate the facing direction of the head, using visual landmarks to distinguish directions. In rats, we investigated whether this visual information is routed through the thalamus to the visual cortex or arrives via the superior colliculus, which is a phylogenetically older and (in rodents) larger pathway. We lesioned the thalamic dorsal lateral geniculate nucleus (dLGN) in rats and recorded the responsiveness of cortical HD cells to visual cues. We found that cortical HD cells had normal tuning curves, but these were slightly more unstable during a trial. Most notably, HD cells in dLGN-lesioned animals showed little ability to distinguish highly distinct cues and none to distinguish more similar cues. These results suggest that directional processing of visual landmarks in mammals requires the geniculo-cortical pathway, which raises questions about when and how visual directional landmark processing appeared during evolution.

Ants use the panoramic skyline as a visual cue during navigation

The VR-based circular tracking movement evaluation system (CES) was developed to quantitatively assess visuomotor control. The virtual stick, a component of the CES, provides visual cues in the virtual environment and haptic feedback when holding the controller. This study examined the effects of stick presence and presentation order on control accuracy for distance, angle, and angular velocity. Twenty-seven participants (12 females; mean age 23.3 ± 2.3 years) performed tasks in the frontal plane followed by the sagittal plane. In each plane, the stick was visible for the first 1-3 revolutions and invisible for the subsequent 4-6 revolutions in the invisible condition, with the reverse order in the visible condition. In the invisible condition, control accuracy with the stick was 1.10 times higher for distance only in the sagittal plane, and 1.13 and 1.09 times higher for angle and angular velocity in the frontal plane, and 1.11 and 1.08 times higher in the sagittal plane. No significant differences were observed in the visible condition. The improved control accuracy when the stick was visible is likely due to enhanced precision in constructing the reference frame, internal models, body coordinates, and effective multisensory integration of visual and haptic information. Such visual information may enable fine control in virtual environment-based applications, including games and surgical simulations.

Animals must routinely deal with barriers as they move through their natural environment. These challenges require directed changes in leg movements and posture performed in the context of ever changing internal and external conditions. In particular, cockroaches use a combination of tactile and visual information to evaluate objects in their path in order to effectively guide their movements in complex terrain. When encountering a large block, the insect uses its antennae to evaluate the object’s height then rears upward accordingly before climbing. A shelf presents a choice between climbing and tunneling that depends on how the antennae strike the shelf; tapping from above yields climbing, while tapping from below causes tunneling. However, ambient light conditions detected by the ocelli can bias that decision. Similarly, in a T-maze turning is determined by antennal contact but influenced by visual cues. These multi-sensory behaviors led us to look at the central complex as a center for sensori-motor integration within the insect brain. Visual and antennal tactile cues are processed within the central complex and, in tethered preparations, several central complex units changed firing rates in tandem with or prior to altered step frequency or turning, while stimulation through the implanted electrodes evoked these same behavioral changes. To further test for a central complex role in these decisions, we examined behavioral effects of brain lesions. Electrolytic lesions in restricted regions of the central complex generated site specific behavioral deficits. Similar changes were also found in reversible effects of procaine injections in the brain. Finally, we are examining these kinds of decisions made in a large arena that more closely matches the conditions under which cockroaches forage. Overall, our studies suggest that CC circuits may indeed influence the descending commands associated with navigational decisions, thereby making them more context dependent.