Audiovisual integration increases the intentional step synchronization of side-by-side walkers

Responses of Domestic Horses and Ponies to Single, Combined and Conflicting Visual and Auditory Cues

Some events in the early history of the oscillating magnetron

ABSTRACTBackground: Phonological cuing in spoken picture naming is widely used to improve word production accuracy in aphasic participants and to investigate word planning processes in psycholinguistic experiments with healthy participants. However, different paradigms and different modalities have been used across studies and populations, with some contradictory results.Aims: Here, we investigate whether and how the modality in which a phonological cue is delivered impacts on word production in picture naming tasks using the exact same paradigm with healthy and with aphasic participants.Methods & Procedure: In experiment 1, behavioural and electroencephalographic event-related potential (ERP) responses to auditory, visual and audiovisual cues in a picture naming task were investigated in healthy participants. In experiment 2, the impact of the same cue modalities was studied on errors and latencies in 15 aphasic participants.Outcomes & Results: The modality in which the cue is delivered influenced phonological facilitation effects in both populations, the audio modality being the most facilitative on production latencies in healthy participants and in participants with mild aphasia. No further gains were obtained by a double modality audiovisual cue on production latencies, whereas the double cues interfered with production accuracy in aphasic participants. ERP results not only were compatible with facilitation of phonological encoding in all experimental conditions but also showed modulation of a late encoding time period for both the visual and audiovisual cues, likely reflecting interference of the visual modality during monitoring or motor speech encoding. The latter observation is indeed coherent with the interference of the audiovisual modality that was observed on error rates in the more severely impaired aphasic participants.Conclusions: These results suggest that unimodal auditory cuing is the facilitatory condition, while visual cues in unimodal or bimodal conditions give rise to both facilitation and interference which manifest behaviourally depending on the balance of these opposing effects.

BackgroundPlanning and executing movements requires the integration of different sensory modalities, such as vision and proprioception. However, neurological diseases like stroke can lead to full or partial loss of proprioception, resulting in impaired movements. Recent advances focused on providing additional sensory feedback to patients to compensate for the sensory loss, proving vibrotactile stimulation to be a viable option as it is inexpensive and easy to implement. Here, we test how such vibrotactile information can be integrated with visual signals to estimate the spatial location of a reach target.MethodsWe used a center-out reach paradigm with 31 healthy human participants to investigate how artificial vibrotactile stimulation can be integrated with visual-spatial cues indicating target location. Specifically, we provided multisite vibrotactile stimulation to the moving dominant arm using eccentric rotating mass (ERM) motors. As the integration of inputs across multiple sensory modalities becomes especially relevant when one of them is uncertain, we additionally modulated the reliability of visual cues. We then compared the weighing of vibrotactile and visual inputs as a function of visual uncertainty to predictions from the maximum likelihood estimation (MLE) framework to decide if participants achieve quasi-optimal integration.ResultsOur results show that participants could estimate target locations based on vibrotactile instructions. After short training, combined visual and vibrotactile cues led to higher hit rates and reduced reach errors when visual cues were uncertain. Additionally, we observed lower reaction times in trials with low visual uncertainty when vibrotactile stimulation was present. Using MLE predictions, we found that integration of vibrotactile and visual cues followed optimal integration when vibrotactile cues required the detection of one or two active motors. However, if estimating the location of a target required discriminating the intensities of two cues, integration violated MLE predictions.ConclusionWe conclude that participants can quickly learn to integrate visual and artificial vibrotactile information. Therefore, using additional vibrotactile stimulation may serve as a promising way to improve rehabilitation or the control of prosthetic devices by patients suffering loss of proprioception.

Multisensory neurons in superior colliculus display inverse effectiveness (IE); responses to multimodal stimuli are greater when the most effective unisensory stimulus produces a weak response (e.g. Meredith & Stein, 1983). Recently, fMRI evidence for inverse effectiveness has been found in superior temporal sulcus (Stevenson & James, 2009), a region implicated in multisensory integration (e.g. Barraclough et al., 2009) and human action perception (e.g. Blake & Shiffrar, 2008). While the relationship between neural IE and multisensory perception and performance remains unclear, and evidence suggests that not all multimodal processes adhere strictly to the IE rule (Holmes, 2007), behavioral studies have shown that the effect of auditory cues on visual processing is greater when the visual signal is relatively weak (e.g., Collignon et al., 2008). Previously, we found that sensitivity to point-light walkers improves when paired with footstep sounds. Because sounds' influence on vision depends upon the strength of the visual signal, we hypothesized that heard footsteps should increase detection sensitivity only when the visual stimulus is weak. To test this hypothesis, we analyzed data from Thomas & Shiffrar (2010) to see whether the detectability of the visual signal determined the degree to which footstep sounds improved detection of point-light walkers. We obtained percent correct for each movie presented in silence (range 33–100%). We then split all movies into two groups based on difficulty. A 3 × 2 mixed measures ANOVA with sound condition as the between-subjects factor (silence, tones, footsteps) and difficulty as the within-subjects factor (easy vs. hard) revealed a main effect of sound, difficulty and a sound X difficulty interaction. Subsequent analyses revealed that footsteps improved visual sensitivity only for “hard” movies, supporting our hypothesis. A follow-up experiment replicated and extended these results.

The phenomenon of crossmodal dynamic visual capture occurs when the direction of motion of a visual cue causes a weakening or reversal of the perceived direction of motion of a concurrently presented auditory stimulus. It is known that there is a perceptual bias towards looming compared to receding stimuli, and faster bimodal reaction times have recently been observed for looming cues compared to receding cues (Cappe et al., 2009). The current studies aimed to test whether visual looming cues are associated with greater dynamic capture of auditory motion in depth compared to receding signals. Participants judged the direction of an auditory motion cue presented with a visual looming cue (expanding disk), a visual receding cue (contracting disk), or visual stationary cue (static disk). Visual cues were presented either simultaneously with the auditory cue, or after 500 ms. We found increased levels of interference with looming visual cues compared to receding visual cues, compared to asynchronous presentation or stationary visual cues. The results could not be explained by the weaker subjective strength of the receding auditory stimulus, as in Experiment 2 the looming and receding auditory cues were matched for perceived strength. These results show that dynamic visual capture of auditory motion in the depth plane is modulated by an adaptive bias for looming compared to receding visual cues.

Role of auditory and visual cues in mutual recognition between ewes and lambs in Merino sheep

The ability to identify pedestrians unobtrusively is essential for smart buildings to provide customized environments, energy saving, health monitoring and security-enhanced services. In this paper, we present an unobtrusive pedestrian identification system by passively listening to people's walking sounds. The proposed acoustic system can be easily integrated with the widely deployed voice assistant devices while providing the context awareness ability. This work focuses on two major tasks. Firstly, we address the challenge of recognizing footstep sounds in complex indoor scenarios by exploiting deep learning and the advanced stereo recording technology that is available on most voice assistant devices. We develop a Convolutional Neural Network-based algorithm and the footstep sound-oriented signal processing schemes to identify users by their footstep sounds accurately. Secondly, we design a "live" footstep detection approach to defend against replay attacks. By deriving the novel inter-footstep and intra-footstep characteristics, we distinguish live footstep sounds from the machine speaker's replay sounds based on their spatial variances. The system is evaluated under normal scenarios, traditional replay attacks and the advanced replays, which are designed to forge footstep sounds both acoustically and spatially. Extensive experiments show that our system identifies people with up to 94.9% accuracy in one footstep and shields 100% traditional replay attacks and up to 99% advanced replay attacks.

Freezing is a disabling symptom in patients with Parkinson's disease. We investigated the effectiveness of a new rehabilitation strategy based on treadmill training associated with auditory and visual cues. Forty Parkinsonian patients with freezing were randomly assigned to two groups: Group 1 underwent a rehabilitation program based on treadmill training associated with auditory and visual cues, while Group 2 followed a rehabilitation protocol using cues and not associated with treadmill. Functional evaluation was based on the Unified Parkinson's Disease Rating Scale Motor Section (UPDRS III), Freezing of Gait Questionnaire (FOGQ), 6-minute walking test (6MWT), gait speed, and stride cycle. Patients in both the groups had significant improvements in all variables considered by the end of the rehabilitation program (all P = 0.0001). Patients treated with the protocol including treadmill, had more improvement than patients in Group 2 in most functional indicators (P = 0.007, P = 0.0004, P = 0.0126, and P = 0.0263 for FOGQ, 6MWT, gait speed, stride cycle, respectively). The most striking result was obtained for 6MWT, with a mean increase of 130 m in Group 1 compared with 57 m in Group 2. Our results suggest that treadmill training associated with auditory and visual cues might give better results than more conventional treatments. Treadmill training probably acts as a supplementary external cue.

In food microbial measurements, when most or very often bacterial counts are below to the limit of quantification (LOQ) or the limit of detection (LOD) in collected food samples, they are either ignored or a specified value is substituted. The consequence of this approach is that it may lead to the over or underestimation of quantitative results. A maximum likelihood estimation (MLE) or Bayesian models can be applied to deal with this kind of censored data. Recently, in food microbiology, an MLE that deals with censored results by fitting a parametric distribution has been introduced. However, the MLE approach has limited practical application in food microbiology as practical tools for implementing MLE statistical methods are limited. We therefore developed a user-friendly MLE tool (called “Microbial-MLE Tool”), which can be easily used without requiring complex mathematical knowledge of MLE but the tool is designated to adjust log-normal distributions to observed counts, and illustrated how this method may be implemented for food microbial censored data using an Excel spreadsheet. In addition, we used two case studies based on food microbial laboratory measurements to illustrate the use of the tool. We believe that the Microbial-MLE tool provides an accessible and comprehensible means for performing MLE in food microbiology and it will also be of help to improve the outcome of quantitative microbial risk assessment (MRA).

That lipreading plays a role in phoneme recognition, even when the acoustic signal alone is phonologically unambiguous, has been concluded from experiments in the perception of discrepant combinations of acoustic and visual speech signals. Little is known about the effect of visual information on explicitly phonetic judgments, the kind of judgments made by trained observers that are the basis for describing the phonological pattern of a language. In this study some isolated vowels, most of them similar to vowels in standard French, were produced in ten random orders by an experienced phonetician. The acoustic signals and frontal views of the lower half of the speaker's face were recorded on video tape. By computer editing, audiovisual stimuli were prepared in which pairs of vowels supposed to differ primarily in rounding were variously combined. Twenty French-speaking speech researchers carried out three tasks: to decide on the rounding of each vowel by sound alone, by sight alone, and by sound when accompanied by matching or discrepant images of the talker. Their summed responses indicate that, despite the instruction to base decisions on the auditory signal, visual evidence of speech activity significantly "perturbed" subjects' rounding judgments. However, the lipreading effect varied greatly across both subjects and vowels. Most subjects judged most vowels strictly on the basis of the auditory information, while for others lipreading exerted paramount influence. Only a small minority responded so as to indicate any integration of discrepant rounding information registered by ear and eye.

Vection is the illusion of self-motion in the absence of real physical movement. The aim of the present study was to analyze how multisensory inputs (visual and auditory) contribute to the perception of vection. Participants were seated in a stationary position in front of a large, curved projection display and were exposed to a virtual scene that constantly rotated around the yaw-axis, simulating a 360° rotation. The virtual scene contained either only visual, only auditory, or a combination of visual and auditory cues. Additionally, simulated rotation speed (90°/s vs. 60°/s) and the number of sound sources (1 vs. 3) were varied for all three stimulus conditions. All participants were exposed to every condition in a randomized order. Data specific to vection latency, vection strength, the severity of motion sickness (MS), and postural steadiness were collected. Results revealed reduced vection onset latencies and increased vection strength when auditory cues were added to the visual stimuli, whereas MS and postural steadiness were not affected by the presence of auditory cues. Half of the participants reported experiencing auditorily induced vection, although the sensation was rather weak and less robust than visually induced vection. Results demonstrate that the combination of visual and auditory cues can enhance the sensation of vection.

Self-perception of body posture and movement is achieved through multi-sensory integration, particularly the utilisation of vision, and proprioceptive information derived from muscles and joints. Disruption to these processes can occur following a neurological accident, such as stroke, leading to sensory and physical impairment. Rehabilitation can be helped through use of augmented visual and auditory biofeedback to stimulate neuro-plasticity, but the effective design and application of feedback, particularly in the auditory domain, is non-trivial. Simple auditory feedback was tested by comparing the stepping accuracy of normal subjects when given a visual spatial target (step length) and an auditory temporal target (step duration). A baseline measurement of step length and duration was taken using optical motion capture. Subjects (n=20) took 20 ‘training’ steps (baseline ±25%) using either an auditory target (950 Hz tone, bell-shaped gain envelope) or visual target (spot marked on the floor) and were then asked to replicate the target step (length or duration corresponding to training) with all feedback removed. Visual cues (mean percentage error=11.5%; SD ± 7.0%); auditory cues (mean percentage error = 12.9%; SD ± 11.8%). Visual cues elicit a high degree of accuracy both in training and follow-up un-cued tasks; despite the novelty of the auditory cues present for subjects, the mean accuracy of subjects approached that for visual cues, and initial results suggest a limited amount of practice using auditory cues can improve performance.

Within high-risk operational environments, expertise has typically been associated with a greater capacity to extract and utilize task-relevant visual cues during situation assessment. However, a limitation of this literature is its exclusive focus on operators’ use of visual cues, even though cues from other modalities (such as auditory cues) are frequently engaged during this assessment process. Arguably, if the capacity for cue utilization is an underlying skill, those operators who have a greater capacity to use visual cues would also have developed a more nuanced repertoire of non-visual cues. Within the context of electricity distribution control, the current study recruited network operators ( N=89) from twelve Australian Distributed Network Service Providers. Using an online experimental platform, participants’ visual cue utilization was assessed using an online situational judgement test (EXPERTise 2.0). Participants also completed the Auditory Readback Task which assessed their capacity to utilize various auditory cues (namely, final rising intonation, fillers, readback accuracy) when recognising nonunderstandings. The results showed a partial relationship between operator capacity for visual and auditory cue utilization. The outcomes of the current research have practical implications for the design of cue-based training interventions to increase the recognition of communication-related errors within distributed environments.

Objective Cocaine is a widely abused drug in the United States and a significant but not well-studied problem in adolescent populations. Cocaine craving seems to be more crucial than other variables related to abstinence because it may lead directly to cocaine use and the recurrence of addiction. Previously, the authors demonstrated with adults that cocaine craving could be induced in a group setting by presenting certain visual and auditory cues on a video monitor. This study investigates the phenomenon of cocaine craving in adolescents. Method Subjects were adolescents aged 14 to 17 years who had abused cocaine and were attending the 6-week adolescent inpatient drug and alcohol treatment program for the South Carolina Department of Mental Health. The subjects viewed a 36-minute videotape with salient cocaine cues. Immediately before and after the exposure, subjects were administered 20 cm analog measures of craving, mood, energy, and wellness. Results Of the 32 subjects completing the study, 25 adolescents reported increased craving upon exposure to video cues (from x=6.74 to x=9.59, P<0.0001). Changes in craving were significantly correlated with changes in mood, energy, and sense of well-being. Conclusions Cocaine craving is a phenomenon present in adolescents and may be induced through audiovisual cues.