A Goal-Directed Visual Perception System Using Object-Based Top–Down Attention

Strong evidence has shown that visual processing based on selective attention is both data- and knowledge-driven. However, most of the previous work mainly focused on the former. We propose in this paper a new selective attention visual computing model based on both of them. The novelty lies in: (1) A structure variable non-uniform sampling method is proposed to separate visual computing into foveal and peripheral channel. (2) A combination of the bottom-up and the top-down selective attention mechanism based on a two-layered pyramid is proposed. The data-driven bottom-up selective attention includes the sequential extraction of feature maps, conspicuity maps, and interesting maps based on the multi-channel filtering and relaxation process. The knowledge driven top-down selective attention is based on distributed associative memory mapping. (3) A movement control mechanism is also proposed in this paper. Perfectly good experiment results on artificial and real times demonstrate the validity of our model.

Divisive normalization models of covert attention commonly use spike rate modulations as indicators of the effect of top-down attention. In addition, an increasing number of studies have shown that top-down attention increases the synchronization of neuronal oscillations as well, particularly in gamma-band frequencies (25–100 Hz). Although modulations of spike rate and synchronous oscillations are not mutually exclusive as mechanisms of attention, there has thus far been little effort to integrate these concepts into a single framework of attention. Here, we aim to provide such a unified framework by expanding the normalization model of attention with a multi-level hierarchical structure and a time dimension; allowing the simulation of a recently reported backward progression of attentional effects along the visual cortical hierarchy. A simple cascade of normalization models simulating different cortical areas is shown to cause signal degradation and a loss of stimulus discriminability over time. To negate this degradation and ensure stable neuronal stimulus representations, we incorporate a kind of oscillatory phase entrainment into our model that has previously been proposed as the “communication-through-coherence” (CTC) hypothesis. Our analysis shows that divisive normalization and oscillation models can complement each other in a unified account of the neural mechanisms of selective visual attention. The resulting hierarchical normalization and oscillation (HNO) model reproduces several additional spatial and temporal aspects of attentional modulation and predicts a latency effect on neuronal responses as a result of cued attention.

Gait recognition and understanding systems have shown a wide-ranging application prospect. However, their use of unstructured data from image and video has affected their performance, e.g., they are easily influenced by multi-views, occlusion, clothes, and object carrying conditions. This paper addresses these problems using a realistic 3-dimensional (3D) human structural data and sequential pattern learning framework with top-down attention modulating mechanism based on Hierarchical Temporal Memory (HTM). First, an accurate 2-dimensional (2D) to 3D human body pose and shape semantic parameters estimation method is proposed, which exploits the advantages of an instance-level body parsing model and a virtual dressing method. Second, by using gait semantic folding, the estimated body parameters are encoded using a sparse 2D matrix to construct the structural gait semantic image. In order to achieve time-based gait recognition, an HTM Network is constructed to obtain the sequence-level gait sparse distribution representations (SL-GSDRs). A top-down attention mechanism is introduced to deal with various conditions including multi-views by refining the SL-GSDRs, according to prior knowledge. The proposed gait learning model not only aids gait recognition tasks to overcome the difficulties in real application scenarios but also provides the structured gait semantic images for visual cognition. Experimental analyses on CMU MoBo, CASIA B, TUM-IITKGP, and KY4D datasets show a significant performance gain in terms of accuracy and robustness.

This paper introduces a method to guide the visual search towards a searched object, analogously to what is performed by the top-down visual attention mechanism. This is done by prioritizing scene descriptors based on their Hamming distance to the descriptors of the target. The proposal has constant space and time complexity in relation to the number of descriptors of the searched object. Moreover, the time overhead to execute the guidance is minimal compared to the recognition time. Experiments indicate that the proposed method is able to reduce the matching time up to 8% of the processing time of classic way of matching.

The visual system employs a sophisticated balance of attentional mechanisms: salient stimuli are prioritized for visual processing, yet observers can also ignore such stimuli when their goals require directing attention elsewhere. A powerful determinant of visual salience is local feature contrast: if a local region differs from its immediate surround along one or more feature dimensions, it will appear more salient. We used high-resolution functional MRI (fMRI) at 7T to characterize the modulatory effects of bottom-up salience and top-down voluntary attention within multiple sites along the early visual pathway, including visual areas V1-V4 and the lateral geniculate nucleus (LGN). Observers viewed arrays of spatially distributed gratings, where one of the gratings immediately to the left or right of fixation differed from all other items in orientation or motion direction, making it salient. To investigate the effects of directed attention, observers were cued to attend to the grating to the left or right of fixation, which was either salient or nonsalient. Results revealed reliable additive effects of top-down attention and stimulus-driven salience throughout visual areas V1-hV4. In comparison, the LGN exhibited significant attentional enhancement but was not reliably modulated by orientation- or motion-defined salience. Our findings indicate that top-down effects of spatial attention can influence visual processing at the earliest possible site along the visual pathway, including the LGN, whereas the processing of orientation- and motion-driven salience primarily involves feature-selective interactions that take place in early cortical visual areas.NEW & NOTEWORTHY While spatial attention allows for specific, goal-driven enhancement of stimuli, salient items outside of the current focus of attention must also be prioritized. We used 7T fMRI to compare salience and spatial attentional enhancement along the early visual hierarchy. We report additive effects of attention and bottom-up salience in early visual areas, suggesting that salience enhancement is not contingent on the observer's attentional state.

Motivated by human eye movement strategies a computer based ’attentional mechanism’ for processing face images is developed that comprises preattentive and attentive processing strategies. Both parts use a common filtering framework based on steerable filters. During the preattentive processing, prominent facial regions like the eye or the mouth region are localized. The selected regions are analyzed in more detail in the attentive processing step. A variety of complex features are derived applying an efficient filter method based on steerable filters.KeywordsBasis FunctionFace ImageAttentive ProcessingSalient RegionSimple FilterThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Multiscale face recognition in cluttered backgrounds based on visual attention

While abundant in biology, foveated vision is nearly absent from computational models and especially deep learning architectures. Despite considerable hardware improvements, training deep neural networks still presents a challenge and constraints complexity of models. Here we propose an end-to-end neural model for foveal-peripheral vision, inspired by retino-cortical mapping in primates and humans. Our model has an efficient sampling technique for compressing the visual signal such that a small portion of the scene is perceived in high resolution while a large field of view is maintained in low resolution. An attention mechanism for performing “eye-movements” assists the agent in collecting detailed information incrementally from the observed scene. Our model achieves comparable results to a similar neural architecture trained on full-resolution data for image classification and outperforms it at video classification tasks. At the same time, because of the smaller size of its input, it can reduce computational effort tenfold and uses several times less memory. Moreover, we present an easy to implement bottom-up and top-down attention mechanism which relies on task-relevant features and is therefore a convenient byproduct of the main architecture. Apart from its computational efficiency, the presented work provides means for exploring active vision for agent training in simulated environments and anthropomorphic robotics.

A bottom-up and top-down attention mechanism has led to the revolutionizing of image captioning techniques, which enables object-level attention for multi-step reasoning over all the detected objects. However, when humans describe an image, they often apply their own subjective experience to focus on only a few salient objects that are worthy of mention, rather than all objects in this image. The focused objects are further allocated in linguistic order, yielding the “object sequence of interest” to compose an enriched description. In this work, we present the Bottom-up and Top-down Object inference Network (BTO-Net), which novelly exploits the object sequence of interest as top-down signals to guide image captioning. Technically, conditioned on the bottom-up signals (all detected objects), an LSTM-based object inference module is first learned to produce the object sequence of interest, which acts as the top-down prior to mimic the subjective experience of humans. Next, both of the bottom-up and top-down signals are dynamically integrated via an attention mechanism for sentence generation. Furthermore, to prevent the cacophony of intermixed cross-modal signals, a contrastive learning-based objective is involved to restrict the interaction between bottom-up and top-down signals, and thus leads to reliable and explainable cross-modal reasoning. Our BTO-Net obtains competitive performances on the COCO benchmark, in particular, 134.1% CIDEr on the COCO Karpathy test split. Source code is available at https://github.com/YehLi/BTO-Net .

Neural mechanisms of selective auditory attention are enhanced by computerized training: Electrophysiological evidence from language-impaired and typically developing children

Understanding bottom-up and top-down visual attention mechanisms related to visual quality perception can be greatly beneficial for the design of effective objective quality metrics. Subjective studies based on eye-movement tracking have been recently published that try to get more insight in these interactions. However, it is still not easy to find coherence across their results, also due to the different methodologies adopted to analyze eye-tracking data. In this paper we propose a robust methodology to measure differences between eye-tracking data collected under different experimental conditions. The proposed method takes into account inter-observer variability and content effects, producing results that give an accurate insight in attention variations.

Irrelevant events in one sensory modality can influence the number of events that are perceived in another modality. Previously, the underlying process of sensory integration was studied in conditions in which participants knew a priori which sensory modality was relevant and which was not. Consequently, (bottom-up) sensory interference and (top-down) selective attention were confounded. We disentangled these effects by measuring the influence of visual flashes on the number of tactile taps that were perceived, and vice versa, in two conditions. In the cue condition, participants were instructed on which modality to report before the bimodal stimulus was presented. In the no-cue condition, they were instructed after stimulus presentation. Participants reported the number of events that they perceived for bimodal combinations of one, two, or three flashes and one, two, or three taps. Our main findings were that (1) in no-cue conditions, the influence of vision on touch was stronger than was the influence of touch on vision; (2) in cue conditions, the integration effects were smaller than those in no-cue conditions; and (3) irrelevant taps were less easily ignored than were irrelevant flashes. This study disentangled previously confounded bottom-up and top-down effects: The bottom-up influence of vision on touch was stronger, but vision was also more easily suppressed by top-down selective attention. We have compared our results qualitatively and quantitatively with recently proposed sensory-integration models.

Models of attention demonstrated the existence of top-down, bottom-up, and history-driven attentional mechanisms, controlled by partially segregated networks of brain areas. However, few studies have examined the specific deficits in those attentional mechanisms in intellectual disability within the same experimental setting. The aim of the current study was to specify the attentional deficits in intellectual disability in top-down, bottom-up, and history-driven processing of multisensory stimuli, and gain insight into effective attentional cues that could be utilized in cognitive training programs for intellectual disability. The performance of adults with mild to moderate intellectual disability (n = 20) was compared with that of typically developing controls (n = 20) in a virtual reality visual search task. The type of a spatial cue that could aid search performance was manipulated to be either endogenous or exogenous in different sensory modalities (visual, auditory, tactile). The results identified that attentional deficits in intellectual disability are overall more pronounced in top-down rather than in bottom-up processing, but with different magnitudes across cue types: The auditory or tactile endogenous cues were much less effective than the visual endogenous cue in the intellectual disability group. Moreover, the history-driven processing in intellectual disability was altered, such that a reversed priming effect was observed for immediate repetitions of the same cue type. These results suggest that the impact of intellectual disability on attentional processing is specific to attentional mechanisms and cue types, which has theoretical as well as practical implications for developing effective cognitive training programs for the target population.

Models of attention demonstrated the existence of top-down, bottom-up, and history-driven attentional mechanisms, controlled by partially segregated networks of brain areas. However, few studies have examined the specific deficits in those attentional mechanisms in intellectual disability within the same experimental setting. The aim of the current study was to specify the attentional deficits in intellectual disability in top-down, bottom-up, and history-driven processing of multisensory stimuli, and gain insight into effective attentional cues that could be utilized in cognitive training programs for intellectual disability. The performance of adults with mild to moderate intellectual disability (n = 20) was compared with that of typically developing controls (n = 20) in a virtual reality visual search task. The type of a spatial cue that could aid search performance was manipulated to be either endogenous or exogenous in different sensory modalities (visual, auditory, tactile). The results identified that attentional deficits in intellectual disability are overall more pronounced in top-down rather than in bottom-up processing, but with different magnitudes across cue types: The auditory or tactile endogenous cues were much less effective than the visual endogenous cue in the intellectual disability group. Moreover, the history-driven processing in intellectual disability was altered, such that a reversed priming effect was observed for immediate repetitions of the same cue type. These results suggest that the impact of intellectual disability on attentional processing is specific to attentional mechanisms and cue types, which has theoretical as well as practical implications for developing effective cognitive training programs for the target population.

Selective visual attention describes the tendency of visual processing to be confined largely to stimuli that are relevant to behavior. It is among the most fundamental of cognitive functions, particularly in humans and other primates for whom vision is the dominant sense. We review recent progress in identifying the neural mechanisms of selective visual attention. We discuss evidence from studies of different varieties of selective attention and examine how these varieties alter the processing of stimuli by neurons within the visual system, current knowledge of their causal basis, and methods for assessing attentional dysfunctions. In addition, we identify some key questions that remain in identifying the neural mechanisms that give rise to the selective processing of visual information.