An unsupervised automatic change detection approach based on visual attention mechanism

A new approach for image segmentation based on visual attention mechanism is proposed. Motivated biologically, this approach simulates the bottom-up human visual selective attention mechanism, extracts early vision features of the image and constructs the saliency map. Multi ple image features such as intensity, color and orientation in multiple scales are extracted to get some feature maps. The phase spectra of the feature maps are analyzed in frequency spectrum domain. Then the corresponding feature saliency maps are constructed in spatial domain and theses feature saliency maps are combined to an integrated saliency map. According to the saliency map, the salient regions in an image are detected. The image is segmented by seperating the salient regions and the backgroud. This approach has been tested on natrual images. Experiment results and quantitative and qualitative analysis have been presented in this paper. The proposed results are consistent with human manual segmentation results. T his proposed approach is effective in computational speed, segmentation result and it is robust to noise .

Selective visual attention can direct our gaze rapidly towards objects of interest in the view. Better coverage of target region for attention can better serve for recognition. A novel method for evaluating how well the attended regions contribute to the recognition of the target based on SIFT descriptors is proposed in this paper. The method is used to evaluate the attended regions extracted by some visual attention mechanisms on real remote sensor images with different geometric and photometric transformations and for different scene types. The evaluation method proposed in this paper give an explicit, accurate and robust expression about the attended region in visual attention mechanisms and we believe this method could be applicable in visual attention mechanisms in the future work.

The concept of inspection robot is born in the strong & smart grid. By studying on the mechanism of human visual attention, this paper presents a method to detect the encroachments on transmission Lines from the perspective of the saLiency-based visual attention system. There are conspicuous diffidence between the encroachments and transmission Lines and it is easy for human to distinguish the encroachments from the transmission Lines because the encroachments were always the saLient objects in the view, consequently, it is a reasonable way to detect the encroachments based on the mechanism of visual attention. The experiment results show that this method can effectively detect and locate different types of encroachments in different natural environments.

Unsupervised change detection(CD) from remotely sensed images is a fundamental challenge when the ground truth for supervised learning is not easily available. Inspired by the visual attention mechanism and multi-level sensation capacity of human vision, we proposed a novel multi-scale analysis framework based on multi-scale visual saliency coarse-to-fine fusion (MVSF) for unsupervised CD in this paper. As a preface of MVSF, we generalized the connotations of scale as four classes in the field of remote sensing (RS) covering the RS process from imaging to image processing, including intrinsic scale, observation scale, analysis scale and modeling scale. In MVSF, superpixels were considered as the primitives for analysing the difference image(DI) obtained by the change vector analysis method. Then, multi-scale saliency maps at the superpixel level were generated according to the global contrast of each superpixel. Finally, a weighted fusion strategy was designed to incorporate multi-scale saliency at a pixel level. The fusion weight for the pixel at each scale is adaptively obtained by considering the heterogeneity of the superpixel it belongs to and the spectral distance between the pixel and the superpixel. The experimental study was conducted on three bi-temporal remotely sensed image pairs, and the effectiveness of the proposed MVSF was verified qualitatively and quantitatively. The results suggest that it is not entirely true that finer scale brings better CD result, and fusing multi-scale superpixel based saliency at a pixel level obtained a higher F1 score in the three experiments. MVSF is capable of maintaining the detailed changed areas while resisting image noise in the final change map. Analysis of the scale factors in MVSF implied that the performance of MVSF is not sensitive to the manually selected scales in the MVSF framework.

To address the issues of low accuracy and significant noise impact in edge continuity detection of art and design images, this paper proposes a visual attention-based edge continuity detection method for art and design images. Determine the edge position points of art and design images using the Laplace operator, and obtain the horizontal and vertical gradients of the image edges by calculating the first-order difference method to determine the amplitude of the image edge gradient; By using the maximum and minimum operations in binary morphology instead of intersection and union operations, the image grayscale is determined, and the HSI color space features are determined to complete the edge feature extraction of art and design images. Using the SUSAN operator to clarify the kernel similarity zone of the edges of art and design images, removing similar edge image pixels, using spatial domain denoising and frequency domain denoising to reduce the edge noise of art and design images, and achieving edge preprocessing of art and design images. Introducing visual attention mechanism to transform the pixel space of edge features in art and design images, performing threshold segmentation on the edges of art and design images, and continuously annotating the segmented edge pixels. Introducing loss function to improve the convergence speed of detection, constructing an art and design image edge continuity detection model based on visual attention mechanism, outputting detection results, and implementing research. The experimental results show that the proposed method has better continuity due to the successful avoidance of noise interference by introducing visual attention mechanisms and other operations. As the number of detected pixels increases, the detection deviation rate of the proposed method remains between 0.02% and 0.03%. The proposed method has a lower detection bias rate and can effectively improve the performance of edge continuity detection in art and design images.

This paper proposed a supervised visual attention mechanism for multimodal neural machine translation (MNMT), trained with constraints based on manual alignments between words in a sentence and their corresponding regions of an image. The proposed visual attention mechanism captures the relationship between a word and an image region more precisely than a conventional visual attention mechanism trained through MNMT in an unsupervised manner. Our experiments on English-German and German-English translation tasks using the Multi30k dataset and on English-Japanese and Japanese-English translation tasks using the Flickr30k Entities JP dataset show that a Transformer-based MNMT model can be improved by incorporating our proposed supervised visual attention mechanism and that further improvements can be achieved by combining it with a supervised cross-lingual attention mechanism (up to +1.61 BLEU, +1.7 METEOR).

ABSTRACTIn this study, we evaluated the effects of topographic correction and gap filling of Landsat Enhanced Thematic Mapper Plus (ETM+) images on the accuracy of forest change detection through a trajectory-based approach. Four types of Landsat time series stacks (LTSS) were generated. These stacks resulted from combinations of topographically corrected and uncorrected imagery combined with gap-filled and unfilled stacks. These combinations of stacks were then used as input into a trajectory-based change detection. The results of change detection from trajectory-based analysis using these LTSS were compared in order to assess the effects of both topographic correction and gap-filling procedures on the ability to detect forest disturbances. The results showed that overall accuracies of change detection were improved after gap filling (10.5% and 7.5%), but were only slightly improved after topographic correction (3.6% and 0.6%). Although the gap-filling process introduced some uncertainty that might have caused false change detection, the number of pixels whose detection of disturbance was enhanced after gap filling exceeded those detecting false change. The results also showed that the topographic correction did not contribute much to improve the change detection in this study area. However, topographic correction has a potential to increase the accuracy of change detection in areas of more rugged terrain and steep slopes. This is because a direct relationship between the slope of the topography with topographic correction and an enhanced detection of disturbance in pixels from year to year was observed in this study. For robust change detection, we recommend that a gap-filling process should be included in the trajectory-based analysis procedures such as the one used in this study where a single image per year is used to characterize change. We also recommend that in areas of rugged terrain, a topographic correction in the image pre-processing should be implemented.

Detecting multiple salient objects in complex scenes is a challenging task. In this paper, we present a novel method to detect salient objects in images. The proposed method is based on the general ‘center-surround’ visual attention mechanism and the spatial frequency response of the human visual system (HVS). The saliency computation is performed in a statistical way. This method is modeled following three biologically inspired principles and compute saliency by two ‘scatter matrices’ which are used to measure the variability within and between two classes, i.e., the center and surrounding regions, respectively. In order to detect multiple salient objects of different sizes in a scene, the saliency of a pixel is estimated via its saliency support region which is defined as the most salient region centered at the pixel. Compliance with human perceptual characteristics enables the proposed method to detect salient objects in complex scenes and predict human fixations. Experimental results on three eye tracking datasets verify the effectiveness of the method and show that the proposed method outperforms the state-of-the-art methods on the visual saliency detection task.

Time-effective detection and recognition of objects of interest in images is still a matter of intensive research in computer vision community because the artificial vision systems fail to outperform the detection results by a human being. The detection problem is complicated when objects of interest have low contrast and various sizes or orientations and can be located on noisy and inhomogeneous background with occlusion occurrences. In many practical applications, the real-time implementation of object detection algorithms in such conditions is a matter of great concern. The results of numerous neurological and psychophysical investigation of human visual system (HVS) indicate that the human vision can successfully cope with these complex situations because of using the visual attention mechanism associated with a model-based image analysis.1,2 The goal of the investigation presented here was not the simulation of human visual perception but the incorporation of its advantageous features into computer vision algorithms. Besides many remarkable properties of HVS like the mentioned model-based visual attention, the HVS has also some disadvantages while detecting and identifying objects. For instance, it is prone to illusions that should be not automatically copied onto an artificial vision system.3

Traditional moving target detection methods usually yield a poor performance when used in through-the-wall radar (TWR) because of the low signal-to-clutter ratio caused by obstacles. In this paper, a new moving target detection method based on the visual attention mechanism is proposed for TWR. The proposed method employs the information stored in the system memory to reduce false alarms while maintain a high detection performance, which is extracted from the TWR multi-frame images. The updating of memory information is based on the principle of visual attention mechanism to fulfill the real-time requirement. Real data collected by the experimental TWR system with multiple-input-multiple-output arrays have been used to validate the efficiency of the proposed method.

With its synoptic view and the repeatability, satellite remote sensing can provide timely, accurate and consistent information about earth’s surface for cost-effective monitoring of environmental changes. In this chapter, recent development in change detection techniques using multitemporal remotely sensed images were reviewed. The chapter covers change detection methods for both optical and SAR images. Various aspects of change detection processes were presented including data preprocessing, change image generation and change detection algorithms such as unsupervised and supervised change detection as well as pixel-based and object-based change detection. The review shows that significant progress has been made in the field of change detection and innovative methods have been developed for change detection using both multitemporal SAR and optical data. Attempts have been made for change detection using multitemporal multisensor/cross-sensor images. The review also identified a number of challenges and opportunities in change detection.

Change detection by comparing two bitemporal images is one of the most fundamental challenges for dynamic monitoring of the Earth surface. In this article, we propose a metric learning-based generative adversarial network (GAN) (MeGAN) to automatically explore seasonal invariant features for pseudochange suppressing and real change detection. To achieve this purpose, a seasonal invariant term is introduced to maximally suppress pseudochanges, whereas the MeGAN explores the transition patterns between adjacent images in a self-learning fashion. Different from the previous works on bitemporal imagery change detection, the proposed MeGAN have the following contributions: 1) it automatically explores change patterns from the complex bitemporal background without human intervention and 2) it aims to maximally exclude pseudochanges from the seasonal transition term and map out real changes efficiently. To our best knowledge, this is the first time we incorporate the seasonal transition term and GAN for change detection between bitemporal images. At last, to demonstrate the robustness of the proposed method, we included two data sets which are the Google Earth data and the Landsat data, for bitemporal change detection and evaluation. The experimental results indicated that the proposed method is able to perform change detection with precision can be as high as 81% and 88% for the Google Earth and Landsat data set, respectively.

Achieving the accurate perception of occluded objects for autonomous vehicles is a challenging problem. Human vision can always quickly locate important object regions in complex external scenes, while other regions are only roughly analysed or ignored, defined as the visual attention mechanism. However, the perception system of autonomous vehicles cannot know which part of the point cloud is in the region of interest. Therefore, it is meaningful to explore how to use the visual attention mechanism in the perception system of autonomous driving. In this paper, we propose the model of the spatial attention frustum to solve object occlusion in 3D object detection. The spatial attention frustum can suppress unimportant features and allocate limited neural computing resources to critical parts of the scene, thereby providing greater relevance and easier processing for higher-level perceptual reasoning tasks. To ensure that our method maintains good reasoning ability when faced with occluded objects with only a partial structure, we propose a local feature aggregation module to capture more complex local features of the point cloud. Finally, we discuss the projection constraint relationship between the 3D bounding box and the 2D bounding box and propose a joint anchor box projection loss function, which will help to improve the overall performance of our method. The results of the KITTI dataset show that our proposed method can effectively improve the detection accuracy of occluded objects. Our method achieves 89.46%, 79.91% and 75.53% detection accuracy in the easy, moderate, and hard difficulty levels of the car category, and achieves a 6.97% performance improvement especially in the hard category with a high degree of occlusion. Our one-stage method does not need to rely on another refining stage, comparable to the accuracy of the two-stage method.

A novel objective quality for image fusion based on structural similarity and visual attention mechanism (VAM) is presented. By giving higher weight to the salient areas in the input images, the quality measure can estimate how much visual meaningful information is preserved in the fused image. The correlation analysis between objective measure and subjective evaluation showed that our measures are more consistent with human subjective evaluation.

Person re-identification, which aims to compare a person of interest as seen in a “probe” camera view to a “gallery” of candidates captured from a camera that does not overlap with the probe one, has increased significant attention in computer vision due to its application in surveillance and security. Various methods utilize the global information as the feature descriptors, which neglect the details in an image and may have a low accuracy of person re-identification and not much attention has been paid to suppressing the background information, which has an influence on person re-identification to some extent. Being aware of these problems, this paper concentrate on the appearance of a person by improving feature descriptors that shed light on a combination framework by fusing the attribute-identity discrimination network with the person discrimination network based on the visual attention mechanism. Experimental results on publicly available image benchmark data sets have demonstrated that the proposed combination framework can achieve competitive performances as compared with state-of-the-art algorithms in terms of accuracy and effectiveness.