Query generation for helper robots to recognize objects

Detecting multiple salient objects in complex scenes is a challenging task. In this paper, we present a novel method to detect salient objects in images. The proposed method is based on the general ‘center-surround’ visual attention mechanism and the spatial frequency response of the human visual system (HVS). The saliency computation is performed in a statistical way. This method is modeled following three biologically inspired principles and compute saliency by two ‘scatter matrices’ which are used to measure the variability within and between two classes, i.e., the center and surrounding regions, respectively. In order to detect multiple salient objects of different sizes in a scene, the saliency of a pixel is estimated via its saliency support region which is defined as the most salient region centered at the pixel. Compliance with human perceptual characteristics enables the proposed method to detect salient objects in complex scenes and predict human fixations. Experimental results on three eye tracking datasets verify the effectiveness of the method and show that the proposed method outperforms the state-of-the-art methods on the visual saliency detection task.

One of the functionalities a service robot should have, to work in a smart environment, is object recognition and handling of objects. Many researchers have attempted to make service robots recognize objects in natural environments but no conventional vision systems can recognize target objects in complex scenes. We built a prototype smart environment in our research building at Korea Institute of Industrial Technology (KITECH) to demonstrate the feasibility that a service robot with few sensors can provide reliable services by interacting with an environment full of smart devices through wireless sensor network communications. In this paper, we address the issues of how to develop an object recognition system for our RoboMaidHome project. Our object recognition system can not only recognize objects with no need of a previous training stage but make the service robot light-weight with minimum hardware and software requirements while working well in a smart home environment. In addition, the matching process for object recognition is simple since only a few image processing techniques are employed. Our object recognition scheme on the basis of MPEG-7 visual descriptors is under development and includes color, texture, and shape. It will also be incorporated into our mobile service robotics system.

Six degrees-of-freedom (6D) object pose estimation plays an important role in pattern recognition of fields such as robotics and augmented reality. However, there are issues with low accuracy and real-time performance of 6D object pose estimation in complex scenes. To address these challenges, in this article, RFF-PoseNet (a 6D object pose estimation network based on robust feature fusion) is proposed for complex scenes. Firstly, a more lightweight Ghost module is used to replace the convolutional blocks in the feature extraction network. Then, a pyramid pooling module is added to the semantic label branch of PoseCNN to fuse the features of different pooling layers and enhance the network’s ability to capture information about objects in complex scenes and the correlations between contextual information. Finally, a pose regression and optimization module is utilized to further improve object pose estimation in complex scenes. Simulation experiments conducted on the YCB-Video and Occlusion LineMOD datasets show that the RFF-PoseNet algorithm can strengthen the correlation of features between different levels and the recognition ability of unclear targets, thereby achieving excellent accuracy and real-time performance, as well as strong robustness.

A pooling-based feature pyramid network for salient object detection

Effective infrared small target detection is still challenging due to small target sizes and the clutter in the background. Unfortunately, many advanced methods do not perform well in preserving and detecting multi-scale objects in complex scenes. We propose an infrared small targets method to suppress the background and adapt the infrared small targets with different sizes. Based on the singular value analysis in the facet model, we propose a multi-derivative descriptor to enhance the targets and suppress various clutter in the dual derivative channels. In the first-order derivative channel, we design four facet kernels with different directions to enhance and preserve the isotropic small targets and suppress the block clutter. In the second-order derivative channel, we use the facet kernel to enhance the center pixels of targets and suppress the band clutter. In order to adapt to the targets with various sizes, we propose a constrained random walker technique, including an adaptive matching algorithm to extract the local regions of each candidate adaptively based on the constraint of size and shape. The experimental results demonstrate that the proposed method can accurately detect multi-scale small targets in complex scenes, resulting in better detection performance than the state-of-the-art methods.

This paper introduces a novel technique for the automatic identification of real world objects in complex scenes. The identification problem requires the comparison of assemblies of image regions with a previously stored view of a known prototype. Shape context representation and matching are employed for recovering point correspondences between the image and the prototype object. This work is inspired from the problem that, a number of ambient conditions such as partial object occlusion and contour distortion, may affect the performance of the matching process and consequently the identification result. In our approach, subtractive clustering is applied in a novel fashion to enable the identification of regions of interest on the complex scene, based on a density metric. In order to increase the robustness of the identifier to mismatches and reduce the computational cost of the process, a selection of the initial suspicious regions is performed. The performance of the identifier has been examined in a great range of image and prototype selections.

Salient object detection in complex scenes and environments is a challenging research topic. Most works focus on RGB-based salient object detection, which limits its performance of real-life applications when confronted with adverse conditions such as dark environments and complex backgrounds. Taking advantage of RGB and thermal infrared(RGBT) images becomes a new research direction for detecting salient objects in complex scenes, since the thermal infrared spectrum provides the complementary information and has been used in many computer vision tasks. However, current research for RGBT salient object detection is limited by the lack of a large-scale dataset and comprehensive benchmark. This work contributes such a RGBT image dataset named VT5000, including 5000 spatially aligned RGBT image pairs with ground truth annotations. VT5000 has 11 challenges collected in different scenes and environments for exploring the robustness of algorithms. With this dataset, we propose a powerful baseline approach, which extracts multilevel features of each modality and aggregates these features of all modalities with the attention mechanism for accurate RGBT salient object detection. To further solve the problem of blur boundaries of salient objects, we also use an edge loss to refine the boundaries. Extensive experiments show that the proposed baseline approach outperforms the state-of-the-art methods on VT5000 dataset and other two public datasets. In addition, we carry out a comprehensive analysis of different algorithms of RGBT salient object detection on VT5000 dataset, and then make several valuable conclusions and provide some potential research directions for RGBT salient object detection. Our new VT5000 dataset is made publicly available at https://github.com/lz118/RGBT-Salient-Object-Detection.

Object recognition based on convex hull alignment

Visual exploration of unknown environments is considered a typical and highly important task in intelligent robotics. Although robots with visual capabilities comparable to human skills (e.g. mushroom-picking robots or bird-viewing robots) are apparently unachievable in the near future, but the concept of robots able to search for known objects in unknown surroundings is one of the ultimate goals for machine vision applications. In the scenarios that are currently envisaged, the expectations should be realistically limited. Nevertheless, one can expect that a robot, after a visual presentation of an object of interest, should be able to “learn” it and, subsequently, to detect the same object in complex scenes which may be degraded by typical effects, i.e. partial visibility of the objects (due to occlusions and/or poor illumination) and their unpredictable locations. The purpose of this chapter is to propose a novel mechanism that is potentially useful (it has been confirmed by promising preliminary results) in such applications. Several theories exist explaining the human perception of objects (e.g. Edelman, 1997). Some researchers promote the importance of multiple model views (e.g. Tarr et al., 1997) others (e.g. Biederman, 1987) postulate viewpoint invariants in the form of shape primitives (geons). From all the theories, however, the practical conclusion is that vision systems detecting objects in a human-like manner should use locally-perceived features as the fundamental tool for matching the scene content to the models of known objects. The idea of using local features (keypoints, local visual saliencies, interest points, characteristic points, corner points – several almost equivalent names exist) in machine vision can be traced back to the 80’s (e.g. Moravec, 1983; Harris & Stephens, 1988). Although stereovision and motion tracking were initially the most typical applications, it was later found that the same approach can be used in more challenging tasks (e.g. matching images in order to detect partially hidden objects). A well-known Harris-Plessey operator (Harris & Stephens, 1988) was combined with local descriptors of detected points to solve object recognition problems in which local features from analysed images are matched against a database of images depicting known objects (e.g. Schmid & Mohr, 1995). The intention was to retrieve images containing arbitrarily rotated and partially occluded objects. Subsequently developed keypoint detectors address the issues of scale changes (this was the weakest point of the original detectors) and perspective distortions. Generally, to achieve

In order to overcome the problems of the small proportion of the object, the variable shooting angle, and height in the UAV aerial image, an anchor-free small object detection algorithm based on the YOLOX network is proposed in this paper. Firstly, aiming at the problem of small object size, a Tri-Head module is proposed to change the size of the detection head to obtain more original information on small objects. Secondly, in order to accurately detect dense continuous small objects in complex scenes, a CBAM-CSP module incorporating the CBAM attention mechanism is proposed to increase the weights for detecting dense continuous regions of interest in images. Then, to alleviate the problems caused by background clutter and pixel blur, BiNet, a jumping multi-scale feature enhancement module, is proposed. This module fully integrates shallow feature information and deep feature information so that detection heads of different scales can obtain enough semantic image information and spatial information. Finally, the experimental results on the VisDrone dataset show that, comparing with the YOLOX network, the mAP is increased by 7.56%±0.06% when the model parameters are reduced by about 5MB, which proves that the proposed algorithm can effectively improve the detection accuracy of small targets.

Open-vocabulary object detection is an innovative computer vision task capable of widely recognizing and localizing various objects in images. Unlike traditional methods, it can handle diverse object categories and is suitable for real-time applications in dynamic environments. Existing methods typically achieve zero-shot detection capabilities by fusing images and text. However, when semantic discrepancies exist between text and images, biased prediction problems arise, diminishing the effectiveness of semantic guidance. To address these issues, we propose a universal open-vocabulary object detection method that leverages foundational models to provide fine-grained semantic guidance for the detection process. We design a multi-level adaptive scene perception algorithm that captures subtle features of target objects in complex scenes, enabling precise separation of background and foreground. Additionally, we introduce the Text-KAN (T-KAN) model, which integrates textual descriptions with image features. By employing learnable activation functions, it resolves dependencies on linear matrices, enhances text interpretability, corrects semantic biases, and achieves precise alignment between images and text at a fine-grained level. We comprehensively evaluate the performance of our proposed method on existing open-vocabulary benchmarks, conducting experiments on the COCO and LVIS datasets. The results demonstrate significant performance gains in detecting novel categories, highlighting the method’s strong generalization capabilities. This work provides valuable insights and references for advancing the field of open-vocabulary object detection.

To accurately detect multi-scale remote sensing objects in complex backgrounds, we propose a novel transformer-based adaptive object detection method. The backbone network of the method is a dual attention vision transformer network that utilizes spatial window attention and channel group attention to capture feature interactions between different objects in complex scenes. We further design an adaptive path aggregation network. In the designed network, CBAM (Convolutional Block Attention Module) is utilized to suppress background information in the fusion paths of different-level feature maps, and new paths are introduced to fuse same-scale feature maps to increase the feature information of the feature maps. The designed network can provide more effective feature information and improve the feature representation capability. Experiments conducted on the three datasets of RSOD, NWPU VHR-10, and DIOR show that the mAP of our method is 96.9%, 96.6%, and 81.7%, respectively, which outperforms compared object detection methods. The experimental results show that our method can detect remote-sensing objects better.

In this paper, a novel saliency detection algorithm is proposed to fuse both the background and foreground information while detecting salient objects in complex scenes. Firstly, we extract background seeds as well as their spatial information from image borders to construct a background-based saliency map. Then, an optimal contour closure is selected as the foreground region according to the first-stage saliency map. The optimal contour closure can provide a preferable description for salient object. We compute a foreground-based saliency map using the selected foreground region and integrate it with the background-based one. Finally, the unified saliency map is further refined to obtain a more accurate result. Experimental results show that the proposed algorithm can achieve favorable performance compared to the state-of-the-art ones.

Compared with RGB images, light field images contain more abundant visual information, which is helpful to accurately detect salient objects in complex scenes. However, most of the existing light field saliency detection methods use single light field data or do not fully consider the differences and complementarities between different light field data, resulting in insufficient multi-modal fusion. To address these issues, a multi-modal feature fusion network is proposed, which makes full use of the rich visual information in the light field images to realize the accurate saliency object detection. The proposed network consists of two parallel subnets, which are used to process the micro-lens image array and all-foucs image respectively. Then the light field refinement module is used to refine the feature map extracted from the micro-lens array stream, and finally the multi-modal feature fusion is realized by the light field attention module to predict saliency objects more accurately. In order to verify the effectiveness of proposed method, extensive comparison with several existing light field saliency detection algorithms is carried on both Lytro-Illum and LFSD datasets. Experimental results show that the proposed method is superior to others in all evaluation metrics on Lytro-Illum dataset, and has desired generalization abilities on LFSD dataset.

Perceptual localization and focus refinement network for RGB-D salient object detection