Does spatialized audio enhance the creation of mental representations?

ABSTRACTSpatial scaling refers to the ability to map distances between different-sized spaces. This ability has been predominantly investigated in sighted individuals using map-reading tasks. Studies with blind individuals using similar tasks were less informative with respect to spatial scaling. Therefore, the current study examined spatial scaling systematically in congenitally blind and blindfolded sighted individuals (N = 46, aged 8.5–45.0 years). Participants were presented with embossed maps containing a target and asked to indicate the similar position in a referent space. Scaling factors were systematically manipulated by varying the sizes between these spaces. Results indicated that blind participants performed less accurately than sighted individuals. After correcting for reversal errors, this difference disappeared. Our results support previous research showing differences between blind and sighted individuals when using maps, while at the same time qualifying these studies by suggesting that these differences may result from blind individuals’ propensity to interchange left-right directions.

Although the literature on blind individuals’ (forensic) speaker identification abilities is quite scarce, a large variety of other auditory abilities has already been investigated thoroughly in blind listeners and sighted controls. In the following section, results of some relevant studies are reported and discussed. Since the identification of a speaker by his or her voice is a complex task, it is important to consider underlying as well as related auditory abilities, which could attribute to a good speaker identification performance, such as speech perception in noise, frequency discrimination or temporal auditory resolution (see section 2.1).

Everyday communication is enriched by the visual environment that listeners concomitantly link to the linguistic input. If and when visual cues are integrated into the mental meaning representation of the communicative setting, is still unclear. In our earlier findings, the integration of linguistic cues (i.e., topic-hood of a discourse referent) reduced discourse updating costs of the mental representation as indicated by reduced sentence-initial processing costs of the non-canonical word order in German. In the present study we tried to replicate our earlier findings by replacing the linguistic cue by a visual attention-capture cue presented below the threshold of perception in order to direct participant’s attention to a depicted referent. While this type of cue has previously been shown to modulate word order preferences in sentence production, we found no effects on sentence comprehension. We discuss possible theory-based reasons for the null effect of the implicit visual cue as well as methodological caveats and issues that should be considered in future research on multimodal meaning integration.

Dual coding of knowledge in the human brain.

Research has demonstrated that attentional cues overlaid on diagrams and animations can help students attend to the relevant areas and facilitate problem solving. In this study we investigate the influence of visual cues and correctness feedback on students’ reasoning as they solve conceptual physics problems containing a diagram. The participants (N=90) were enrolled in an algebra-based physics course and were individually interviewed. During each interview students solved four problem sets each containing an initial problem, six isomorphic training problems, and a transfer problem. The cued conditions saw visual cues on the training problems, and the feedback conditions were told if their responses (answer and explanation) were correct or incorrect. We found that visual cues and correctness feedback significantly improves students’ abilities to solve the training and transfer problems.

Direction and distance deficits in path integration after unilateral vestibular loss depend on task complexity

Stereophonic audio, especially binaural audio, plays an essential role in immersive viewing environments. Recent research has explored generating visually guided stereophonic audios supervised by multi-channel audio collections. However, due to the requirement of professional recording devices, existing datasets are limited in scale and variety, which impedes the generalization of supervised methods in real-world scenarios. In this work, we propose PseudoBinaural, an effective pipeline that is free of binaural recordings. The key insight is to carefully build pseudo visual-stereo pairs with mono data for training. Specifically, we leverage spherical harmonic decomposition and head-related impulse response (HRIR) to identify the relationship between spatial locations and received binaural audios. Then in the visual modality, corresponding visual cues of the mono data are manually placed at sound source positions to form the pairs. Compared to fully-supervised paradigms, our binaural-recording-free pipeline shows great stability in cross-dataset evaluation and achieves comparable performance under subjective preference. Moreover, combined with binaural recordings, our method is able to further boost the performance of binaural audio generation under supervised settings <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .

Spatial navigation in the absence of vision has been investigated from a variety of perspectives and disciplines. These different approaches have progressed our understanding of spatial knowledge acquisition by blind individuals, including their abilities, strategies, and corresponding mental representations. In this review, we propose a framework for investigating differences in spatial knowledge acquisition by blind and sighted people consisting of three longitudinal models (i.e., convergent, cumulative, and persistent). Recent advances in neuroscience and technological devices have provided novel insights into the different neural mechanisms underlying spatial navigation by blind and sighted people and the potential for functional reorganization. Despite these advances, there is still a lack of consensus regarding the extent to which locomotion and wayfinding depend on amodal spatial representations. This challenge largely stems from methodological limitations such as heterogeneity in the blind population and terminological ambiguity related to the concept of cognitive maps. Coupled with an over‐reliance on potential technological solutions, the field has diffused into theoretical and applied branches that do not always communicate. Here, we review research on navigation by congenitally blind individuals with an emphasis on behavioral and neuroscientific evidence, as well as the potential of technological assistance. Throughout the article, we emphasize the need to disentangle strategy choice and performance when discussing the navigation abilities of the blind population. WIREs Cogn Sci 2016, 7:37–58. doi: 10.1002/wcs.1375For further resources related to this article, please visit the WIREs website.

Speech perception relies on auditory and visual cues and there are strong links between speech perception and production. We aimed to evaluate the role of auditory and visual modalities on speech perception and production in adults with impaired hearing or sight versus those with normal hearing and sight. We examined speech perception and production of three isolated vowels (/i/, /y/, /u/), which were selected based on their different auditory and visual perceptual saliencies, in 12 deaf adults who used one or two cochlear implants (CIs), 14 congenitally blind adults, and 16 adults with normal sight and hearing. The results showed that the deaf adults who used a CI had worse vowel identification and discrimination perception and they also produced vowels that were less typical or precise than other participants. They had different tongue positions in speech production, which possibly partly explains the poorer quality of their spoken vowels. Blind individuals had larger lip openings and smaller lip protrusions for the rounded vowel and unrounded vowels, compared to the other participants, but they still produced vowels that were similar to those produced by the adults with normal sight and hearing. In summary, the deaf adults, even though they used CIs, had greater difficulty in producing accurate vowel targets than the blind adults, whereas the blind adults were still able to produce accurate vowel targets, even though they used different articulatory strategies.

Soil moisture (SM) inversion using Global Navigation System Reflectometry techniques has garnered significant attention in recent years. However, current Cyclone Global Navigation Satellite System (CYGNSS) SM products generally suffer from low spatial resolution, which limits their application in regional and local scale studies. Therefore, this study proposed a SM downscaling method based on multi-model fusion, aimed at enhancing the spatial resolution of CYGNSS data from 36 km to 3 km. The method integrates CYGNSS observations, SM active passive (SMAP) (36 km) SM data, and auxiliary environmental variables to construct four individual machine learning models: random forest, light gradient boosting machine, categorical boosting, and gradient boosting trees as well as the Stacking fusion model. The results of each model downscaling SM comparison with International SM Network stations show that the Stacking model outperforms the single model in correlation coefficient (R), root mean square error, and mean absolute error, while simultaneously exhibiting superior inversion accuracy. In the spatial correlation analysis using moderate resolution imaging spectroradiometer enhanced vegetation index and VIIRS land surface temperature products, the Stacking model results have the highest correlation and more detailed spatial distribution. In addition, the Stacking model downscaled SM also showed good spatial consistency and accuracy in the comparison and verification with SMAP/Sentinel-1 SM (3 km). The results show that the Stacking model, which integrates multiple machine learning algorithms, significantly improves the spatial resolution of the CYGNSS SM products and effectively improves the inversion accuracy, which is of certain reference value for the development of high-resolution SM products.

ABSTRACTIntroduction: Cross-modal interactions improve the processing of external stimuli, particularly when an isolated sensory modality is impaired. When information from different modalities is integrated, object recognition is facilitated probably as a result of bottom-up and top-down processes. The aim of this study was to investigate the potential effects of cross-modal stimulation in a case of simultanagnosia. Method: We report a detailed analysis of clinical symptoms and an 18F-fluorodeoxyglucose (FDG) brain positron emission tomography/computed tomography (PET/CT) study of a patient affected by Balint’s syndrome, a rare and invasive visual–spatial disorder following bilateral parieto-occipital lesions. An experiment was conducted to investigate the effects of visual and nonvisual cues on performance in tasks involving the recognition of overlapping pictures. Four modalities of sensory cues were used: visual, tactile, olfactory, and auditory. Results: Data from neuropsychological tests showed the presence of ocular apraxia, optic ataxia, and simultanagnosia. The results of the experiment indicate a positive effect of the cues on the recognition of overlapping pictures, not only in the identification of the congruent valid-cued stimulus (target) but also in the identification of the other, noncued stimuli. All the sensory modalities analyzed (except the auditory stimulus) were efficacious in terms of increasing visual recognition. Conclusions: Cross-modal integration improved the patient’s ability to recognize overlapping figures. However, while in the visual unimodal modality both bottom-up (priming, familiarity effect, disengagement of attention) and top-down processes (mental representation and short-term memory, the endogenous orientation of attention) are involved, in the cross-modal integration it is semantic representations that mainly activate visual recognition processes. These results are potentially useful for the design of rehabilitation training for attentional and visual–perceptual deficits.

Sequential movements rely on two information sources: external sensory cues and internal memory representations. Although often both sources jointly drive sequential behavior, previous research has primarily examined them in isolation. To address this, we trained participants (n = 26, 15F) to perform sequences of rapid finger presses in response to numerical cues. Sensory influence was measured by varying the number of visible cues, and memory influence was determined by comparing repeating and random sequences. Early in learning, participants integrated sensory and memory information: repeating sequences were performed more quickly when more cues were visible. After learning, when repeating sequences were predictable with certainty, participants relied solely on memory and ignored sensory cues. However, when this certainty was manipulated by introducing occasional violations within repeating sequences, participants reverted to integrating memory with sensory cues. We propose a computational model that successfully predicted both speed and accuracy of individual presses. Critically, this model relied on the assumption that multiple movements are planned independently of each other. This independence assumption was then validated by examining response patterns to isolated violations in repeating sequences. Finally, we provide evidence into how sequence memories can be flexibly deactivated and reactivated in response to these violations. Together, these results reveal how brain dynamically integrates sensory and memory information to produce sequences of movements.

Self-orientation perception refers to our perceived self-orientation relative to gravity. An internal representation of self-orientation is derived from sensory cues indicating either directly or indirectly the direction of gravity relative to the body. The internal representation can be measured in the laboratory or clinic using visual or haptic measures, or even the body itself. However, each measure is affected differently by the availability of cues and the observer’s actual orientation relative to gravity, suggesting multiple, simultaneous representations of gravity. Visual, vestibular, somatosensory and proprioceptive cues are combined by multisensory integration to provide the most reliable estimates. Multisensory integration provides a robust perception of self-orientation for adults but means that children have a much lower precision in judging vertical before multisensory integration mechanisms are mature. The neurophysiological basis of the perception of self-orientation is a network of brain areas reflecting the multisensory processes that underlie it. This network provides some redundancy that can be exploited for potential patient recovery. Future work will perfect models for predicting perceived self-orientation in ever more challenging situations and how we can improve performance of pilots, divers and astronauts as they explore new situations and new gravity fields, and improve how we, and especially older people, can continue to enjoy our lifelong dance with gravity.

Studying how sensory signals from different sources (sensory cues) are integrated within or across multiple senses allows us to better understand the perceptual computations that lie at the foundation of adaptive behaviour. As such, determining the presence of precision gains – the classic hallmark of cue combination – is important for characterising perceptual systems, their development and functioning in clinical conditions. However, empirically measuring precision gains to distinguish cue combination from alternative perceptual strategies requires careful methodological considerations. Here, we note that the majority of existing studies that tested for cue combination either omitted this important contrast, or used an analysis approach that, unknowingly, strongly inflated false positives. Using simulations, we demonstrate that this approach enhances the chances of finding significant cue combination effects in up to 100% of cases, even when cues are not combined. We establish how this error arises when the wrong cue comparator is chosen and recommend an alternative analysis that is easy to implement but has only been adopted by relatively few studies. By comparing combined-cue perceptual precision with the best single-cue precision, determined for each observer individually rather than at the group level, researchers can enhance the credibility of their reported effects. We also note that testing for deviations from optimal predictions alone is not sufficient to ascertain whether cues are combined. Taken together, to correctly test for perceptual precision gains, we advocate for a careful comparator selection and task design to ensure that cue combination is tested with maximum power, while reducing the inflation of false positives.

Multimodality often exhibits synergistic effects: each modality compliments and compensates for other modalities in transferring coherent, unambiguous, and enriched information for higher interaction efficiency and improved sense of presence. In this paper, we explore one such phenomenon: a positive interaction among the geometric field of view, proprioceptive interaction, and tactile feedback. We hypothesize that, with proprioceptive interaction and tactile feedback, the geometric field of view and thus visibility can be increased such that it is larger than the physical field of view, without causing a significant distortion in the user's distance perception. This, in turn, would further help operation of the overall multimodal interaction scheme as the user is more likely to receive the multimodal feedback simultaneously. We tested our hypothesis with an experiment to measure the user's change in distance perception according to different values of egocentric geometric field of view and feedback conditions. Our experimental results have shown that, when coupled with physical interaction, the GFOV could be increased by up to 170 percent of the physical field of view without introducing significant distortion in distance perception. Second, when tactile feedback was introduced, in addition to visual and proprioceptive cues, the GFOV could be increased by up to 200 percent. The results offer a useful guideline for effectively utilizing of modality compensation and building multimodal interfaces for close range spatial tasks in virtual environments. In addition, it demonstrates one way to overcome the shortcomings of the narrow (physical) fields of views of most contemporary HMDs.