Capacity limits for the detection of changing visual features

Looking, language, and memory: Bridging research from the visual world and visual search paradigms

Visual attention and object recognition are two critical cognitive functions that significantly influence our perception of the world. While these neural processes converge on the temporal cortex, the exact nature of their interactions remains largely unclear. Here, we systematically investigated the interplay between visual attention and object feature coding by training macaques to perform a free-gaze visual search task using natural face and object stimuli. With a large number of units recorded from multiple brain areas, we discovered that units exhibiting visual feature coding displayed a distinct attentional response profile and functional connectivity compared to units not exhibiting feature coding. Attention directed towards search targets enhanced the pattern separation of stimuli across brain areas, and this enhancement was more pronounced for units encoding visual features. Our findings suggest two stages of neural processing, with the early stage primarily focused on processing visual features and the late stage dedicated to processing attention. Importantly, feature coding in the early stage could predict the attentional effect in the late stage. Together, our results suggest an intricate interplay between visual feature and attention coding in the primate brain, which can be attributed to the differential functional connectivity and neural networks engaged in these processes.

Visual attention of construction workers is closely related to their safety performance. Identifying and understanding safety signs on workplace effectively is beneficial to improve visual attention. This study focuses on exploring the influencing mechanism of construction safety sign features on visual attention of construction workers using the eye-tracking technology, in order to improve visual attention and workplace safety performance through optimizing the construction safety signs. A theoretical model of influencing mechanism of safety sign features on visual attention was constructed based on visual information processing theory. To verify the theoretical model, an experiment was conducted as follows: 28 pictures of safety signs including visual and cognitive features were shown on the computer screen, then eye movement data from 41 subjects was obtained using EyeLink1000 Plus. Statistical test methods were employed to analyze the relationship between safety sign features and eye-tracking metrics. The statistical results of theoretical model indicate that, among visual features, red and rectangular safety signs can reduce cognitive load of first fixation, green signs can reduce cognitive difficulties, however visual attention is not closely related to auxiliary words. Among the cognitive features, unfamiliar signs require more cognitive effort, while no significant difference exists in visual attention of different levels of concreteness and sematic closeness. This study provides theoretical and practical basis for improving construction workers’ visual attention through optimizing visual and cognitive features of construction safety signs.

To interact with our environment, sensory signals must be translated into perceptions that guide our behavior. However, our noisy, crowded, and dynamic environment provides too much information to process. As a result, our brain cannot translate all sensory signals into percepts. Thus, our brain, processing the information gathered by our sensory organs, must make reasonable assumptions about the world that can then guide our perception. These assumptions do not reflect the physical reality and are modulated by different cognitive processes such as perceptual history, context, attentional deployment, or relevance. Therefore, we might misperceive certain aspects of our environment, leading to perceptual illusions or hallucinations that can misguide our behavior. Visual illusions are widespread in primates, albeit other sensory modalities are also prone to illusions. While illusions can result from visual processing in the retina or V1, others result from cortical processes that directly guide our attention, perception, and behavior. For example, the perceived coupling of visual features, like an object's color and motion direction, can be incorrect, creating perceptual illusions. Crucially, it is still unknown how the visual information processed in different areas of the visual cortex, like color and motion features, is combined into objects that make up a human’s perception. During stable visual input, physical and perceived stimulus features of illusion-inducing stimuli can either match in correctly bound percepts or differ in misbound percepts. This dichotomy might help to reveal how and where the processing of physical stimulus information stops and perception begins. To contribute to closing this knowledge gap, I investigated the misbinding of color and motion features in the peripheral visual field of humans and monkeys. The main goal of this dissertation was to examine the neural correlates of visual feature binding and misbinding in single neurons of the monkey visual cortex area MT. For this, we first investigated the characteristics of a stimulus that induces misbinding of peripheral visual features in three human psychophysics experiments. One monkey was trained on the same task design used in the third human study, but training could not be completed as experiments had to be stopped when the animal experimentation permit expired. Unfortunately, we could not observe misbound percepts in the monkey’s responses to the stimuli used in the training. Neural recordings were conducted in the motion-selective cortex area MT during the training, but we could not observe any neural correlates relating to misbound features. To improve the spatial accuracy of neural recordings and the flexibility of their positioning, we developed a new camera-based method to navigate electrodes to the target area in three dimensions. Furthermore, I contributed to a variation of this method that allows the precise positioning of microdialysis probes in the brain using custom 3D-printed recording chamber insets that guide the probe to its target location.

The ability to encode, store, and retrieve visually presented objects is referred to as visual working memory (VWM). Although crucial for many cognitive processes, previous research reveals that VWM strictly capacity limited. This capacity limitation is behaviorally observable in the set size effect: the ability to successfully report items in VWM asymptotes at a small number of items. Research into the neural correlates of set size effects and VWM capacity limits in general largely focus on the maintenance period of VWM. However, we previously reported that neural resources allocated to individual items during VWM encoding correspond to successful VWM performance. Here we expand on those findings by investigating neural correlates of set size during VWM encoding. We hypothesized that neural signatures of encoding-related VWM capacity limitations should be differentiable as a function of set size. We tested our hypothesis using High Density Electroencephalography (HD-EEG) to analyze frequency components evoked by flickering target items in VWM displays of set size 2 or 4. We found that set size modulated the amplitude of the 1st and 2nd harmonic frequencies evoked during successful VWM encoding across frontal and occipital-parietal electrodes. Frontal sites exhibited the most robust effects for the 2nd harmonic (set size 2 > set size 4). Additionally, we found a set-size effect on the induced power of delta-band (1–4 Hz) activity (set size 2 > set size 4). These results are consistent with a capacity limited VWM resource at encoding that is distributed across to-be-remembered items in a VWM display. This resource may work in conjunction with a task-specific selection process that determines which items are to be encoded and which are to be ignored. These neural set size effects support the view that VWM capacity limitations begin with encoding related processes.

Salient region of the image, which is composed of salient or interest points, is the most informative part of the image. In this paper, a saliency-based bottom-up visual attention computational model motivated by visual physiological experimental results is used to detect salient region and extract salient points of images. Meanwhile, a method to select number of the salient points to be extracted for each image is presented. Two salient visual features based on the visual attention model were proposed for image retrieval. One feature is the ldquoattention histogramrdquo, which only counts the frequencies of a visual feature in the salient region of the image. The other is the ldquosalient image signature histogram and spatial FOAs(focus of attention) anglogramrdquo, which codes both the local properties around salient points of the image and the spatial information of FOAs. Image retrieval experiments were carried out to evaluate the proposed features. For comparison, traditional global histogram was also used in the experiments. Preliminary experimental results showed that the proposed visual attention-based salient features can achieve encouraging retrieval results.

Application research of interior space design elements based on image processing

Face-to-face interactions are part of everyday life, ranging from family to working in teams and to global communities. Social psychologists have long studied these interactions with the aim of understanding behavior, motivations, and emergence of interaction patterns. An organization is environment rich in daily interactions including structured periodic meetings, planning, brainstorming, negotiations, decision-making and informal gatherings and leaders play a key role in many of them. Leader face problems, propose solutions, make decisions, and often are the main source of inspiration of the employees. Identifying emergent leaders at early stages in organizations is a key issue in organizational behavioral research, and a new problem in social computing. The study of this phenomenon requires sensing of natural face-to-face interactions, automatic extraction of behavioral cues and reliable machine learning algorithms to identify emergent leaders. In this thesis we present a computational approach to analyze emergence of leadership in small groups using multimodal audio and visual features. In the computational framework, we first present an analysis on how an emergent leader is perceived in newly formed, small groups. We present the ELEA (Emergent LEadership Analysis) corpus collected with the aim of analyzing emergence of leaders. We propose to analyze emergent leaders, using a variety of nonverbal cues studied in social psychology and automatically extracted from audio and video streams. Our analysis address how the emergent leader is perceived by his/her peers in terms of speaking and visual active, and its relation with the most dominant person (including external observers’ perception). We then propose to investigate which individual nonverbal channel (or combination of features from different channels) provides better inferences of the emergent leader and related conceptsusing unsupervised and supervised methods. We use a supervised collective approach which adds relational information to the nonverbal cues and compare its performance, with the performance of supervised (non-collective) and unsupervised methods. We also propose to capture the social visual attention patterns from automatically extracted features from video, in order to analyze who receives or gives the largest amount of visual attention in the group. Finally, with the aim of understanding who receives the largest amount of visual attention while speaking and who has the highest dominance ratio (i.e., many occurrences of looking at others while speaking and few occurrences of looking at others while not speaking). We synchronize the audio and video streams to capture the speaking and attention activity patterns. We end our analysis exploring the impact of the verbal content (language style) in the interactions and its influence in the perception of emergent leaders. For the language style analysis, we propose to compute word categories extracted from manual transcriptions of the discussions as well as from automatically detected keywords. We propose to use a supervised method to obtain the relevant features, and to use only the top word categories to predict the emergent leader and related concepts in each group. We then propose to differentiate word categories, between highly context-related and context-free, to explore the feasibility to infer the emergent leader in a fully automatic approach from the context-free language style. This dissertation address an audio and visual analysis of the ubiquitous phenomenon of emergent leadership in a fully automatic computational approach from face-to-face interactions. The nonverbal behavioral analysis is inspired in previous works on social psychology in the context of emergent leadership and related concepts. The automatically extracted nonverbal features are modeled to feed state-of-the-art machine learning techniques in order to infer emergent leaders.

In this paper we explore automatic genre labelling of motion picture previews using audio-visual features present in movie trailers and the focus is on fusion techniques (early fusion and late fusion) and the resultant improvement on classification accuracy. This paper proposes a novel combination of deep learned features (from a pretrained VGG-16 model) obtained using a state-of-the-art shot detector and hand-crafted audio features. This combination of features and an associated comparison of early and late fusion with these features has not been attempted in the literature before. Furthermore, two popular fusion techniques and three distinct classification algorithms are investigated to determine the optimal fusion technique and classifier combination. The study uses a subset of the LMTD-9 movie trailer dataset with selected genres (action, comedy, drama and horror). The best performing low-level audio features are comprised of timbre features extracted using the MIRtoolbox followed by standalone mel-frequency cepstral coefficients. The best performing high-level audio feature is tonality. Audio features are augmented by visual features extracted using a pre-trained convolutional neural network (VGG-16). Feature fusion (early and late fusion) methods are investigated together with classification methods such as extreme gradient boosting, support vector machine and a neural network. Evaluation metrics such as precision, recall, confusion matrices and F1 score are used to measure classification accuracy. Early fusion methods outperform late fusion methods with a classification performance gain of approximately 10% for a four class classification problem. The best classification performance for early fusion obtained with a support vector machine is (73.12% accuracy), followed by the extreme gradient boosting classifier (69.37% accuracy) and neural network classifier (67.50% accuracy), whereas chance is 25%. It is shown that superior classification performance can be achieved by employing early feature fusion of low-level audio descriptors, high-level audio descriptors and high-level visual feature descriptors together with suitable classifiers.

To bridge the semantic gap between low-level visual features and high-level semantic concepts, this paper puts forward a novel semi-supervised learning framework of combining visual and keyword features. We assume all of images in the database have been annotated. In this framework, the visual space graph (VSG) and keyword space graph (KSG) are first constructed by means of manifold-based algorithm to explore the submanifold of Visual space and keyword space. Then, semantic space graph (SSG) is constructed by using neighbor information of KSG and VSF. Afterwards, the VSG and KSG are modified and unified by propagating semantic matrix. Finally, we propose a flexible ranking formula and introduce different relevance feedback methods of different query styles. Experimental results on COREL 5000 images show our method improves image retrieval performance from all aspects.

In content based image retrieval (CBIR) process, every image has been represented in a compact set of local visual features i.e. color, texture, and/or shape of images. This set of local visual features is known as feature vector. In the CBIR process, feature vectors of images have been used to represent or to identify similar images in adequate way. As a result, feature vector construction has always been considered as an important issue since it must reflect proper image semantics using minimal amount of data. The proposed CBIR scheme is based on the combination of color and texture features. In this work initially, we have converted the given RGB image into HSV color image. Subsequently, we have considered H (hue), S (saturation), and V (intensity) components for extraction of visual image features. The texture features have been extracted from the V component of the image using 2D dual-tree complex wavelet transform (2D DT-CWT) where it analyzes the textural patterns in six different directions i.e. ±15\(^{\circ }\), ±45\(^{\circ }\), and ±75\(^{\circ }\). At the same time, we have computed the probability histograms of H and S components of the image respectively and subsequently those are divided into non-uniform bins based on cumulative probability for extraction of color based features. So, in this work both the color and texture features have been extracted simultaneously. Finally, the obtained features have been concatenated to attain the final feature vector and same is considered in image retrieval process. We have tested the novelty and performance of the proposed work in two Corel, two objects, and, a texture image datasets. The experimental results reveal the acceptable retrieval performances for different types of datasets.

The find the key paradigm is a digital psychology experiment paradigm which is a modification of the visual search paradigm. This paradigm is intended to focus on one of the pre-attentive visual attentions where the previous visual search used various pre-attentive visual attention features. While this paradigm focuses on finding out how the whole object can affect the search time of visual attention, for this reason the Find the Key paradigm was developed. This study used 116 individuals with a range of 18 to 26 years. Based on the results of the ANOVA analysis, it was found that there were significant differences from the four difficulty levels of the Find the Key paradigm, with search time increasing at more difficult levels. Therefore the number of types of distractor objects can affect visual attention.

Lateral visually-guided saccades were studied electro-oculographically in 40 patients with progressive supranuclear palsy (PSP) in order to increase understanding of the pathophysiology of the oculomotor syndrome and the suprareticular control of saccades. In the two main paradigms used, the central fixation point was removed just before the appearance of a lateral target, the subject being told either to look at the target (gap task) or, particularly to test frontal lobe influence, in the direction opposite to the target (antisaccade task). Saccade latency distribution in the gap task was significantly more scattered in patients as compared with 40 normal control subjects, although mean latencies did not differ greatly. In particular, many patients had latencies shorter or longer than those of control subjects. In the PSP group, the percentage of errors in the antisaccade task (misdirected saccades made towards the target) was strongly correlated with latency in the gap task, high percentages of errors corresponding to short latencies. There was also a correlation between latency and the frontal dysfunction evaluated by neuropsychological tests. These correlations suggest that short latencies could result from a severe impairment of the frontal inhibitory system involved in saccade initiation. Long latencies would result from damage to diverse excitatory suprareticular pathways (such as the superior colliculus) or to circuits responsible for shifts in visual attention. Lastly, 20 patients were followed longitudinally (for 3-12 months). The initially great intersubject variability of saccade latency in the gap task significantly decreased over a period of several months. This finding could result from a secondary balance between the impairment of the frontal inhibitory system and that of the diverse excitatory pathways.

Human brain activity associated with audiovisual perception and attention

In three experiments, the efficiency in detecting fear-relevant and fear-irrelevant visual stimuli are compared. A visual search paradigm is used where participants are presented with matrices of different sizes (4 objects/9 objects) and must determine whether all objects are taken from the same category or whether there is a discrepant one. Results from all experiments were consistent with the threat-superiority effect. Participants were quicker when the target was threatening than when it was not. Other indicators confirmed that the detection of threatening targets involves more efficient processes (reduced slopes, absence of position effects). A crucial aspect of these experiments was the comparison of evolutionary-relevant (snakes, spiders, etc.) and modern (guns, syringes, etc.) threats. The threat-superiority effect was repeatedly found for both types of target. Stronger effects were sometimes observed for modern than for evolutionary-relevant threats. The implications for evolutionary explanations of the effect of fear on visual attention are discussed.