Object recognition based on convex hull alignment

One of the functionalities a service robot should have, to work in a smart environment, is object recognition and handling of objects. Many researchers have attempted to make service robots recognize objects in natural environments but no conventional vision systems can recognize target objects in complex scenes. We built a prototype smart environment in our research building at Korea Institute of Industrial Technology (KITECH) to demonstrate the feasibility that a service robot with few sensors can provide reliable services by interacting with an environment full of smart devices through wireless sensor network communications. In this paper, we address the issues of how to develop an object recognition system for our RoboMaidHome project. Our object recognition system can not only recognize objects with no need of a previous training stage but make the service robot light-weight with minimum hardware and software requirements while working well in a smart home environment. In addition, the matching process for object recognition is simple since only a few image processing techniques are employed. Our object recognition scheme on the basis of MPEG-7 visual descriptors is under development and includes color, texture, and shape. It will also be incorporated into our mobile service robotics system.

Detecting multiple salient objects in complex scenes is a challenging task. In this paper, we present a novel method to detect salient objects in images. The proposed method is based on the general ‘center-surround’ visual attention mechanism and the spatial frequency response of the human visual system (HVS). The saliency computation is performed in a statistical way. This method is modeled following three biologically inspired principles and compute saliency by two ‘scatter matrices’ which are used to measure the variability within and between two classes, i.e., the center and surrounding regions, respectively. In order to detect multiple salient objects of different sizes in a scene, the saliency of a pixel is estimated via its saliency support region which is defined as the most salient region centered at the pixel. Compliance with human perceptual characteristics enables the proposed method to detect salient objects in complex scenes and predict human fixations. Experimental results on three eye tracking datasets verify the effectiveness of the method and show that the proposed method outperforms the state-of-the-art methods on the visual saliency detection task.

Six degrees-of-freedom (6D) object pose estimation plays an important role in pattern recognition of fields such as robotics and augmented reality. However, there are issues with low accuracy and real-time performance of 6D object pose estimation in complex scenes. To address these challenges, in this article, RFF-PoseNet (a 6D object pose estimation network based on robust feature fusion) is proposed for complex scenes. Firstly, a more lightweight Ghost module is used to replace the convolutional blocks in the feature extraction network. Then, a pyramid pooling module is added to the semantic label branch of PoseCNN to fuse the features of different pooling layers and enhance the network’s ability to capture information about objects in complex scenes and the correlations between contextual information. Finally, a pose regression and optimization module is utilized to further improve object pose estimation in complex scenes. Simulation experiments conducted on the YCB-Video and Occlusion LineMOD datasets show that the RFF-PoseNet algorithm can strengthen the correlation of features between different levels and the recognition ability of unclear targets, thereby achieving excellent accuracy and real-time performance, as well as strong robustness.

A pooling-based feature pyramid network for salient object detection

Effective infrared small target detection is still challenging due to small target sizes and the clutter in the background. Unfortunately, many advanced methods do not perform well in preserving and detecting multi-scale objects in complex scenes. We propose an infrared small targets method to suppress the background and adapt the infrared small targets with different sizes. Based on the singular value analysis in the facet model, we propose a multi-derivative descriptor to enhance the targets and suppress various clutter in the dual derivative channels. In the first-order derivative channel, we design four facet kernels with different directions to enhance and preserve the isotropic small targets and suppress the block clutter. In the second-order derivative channel, we use the facet kernel to enhance the center pixels of targets and suppress the band clutter. In order to adapt to the targets with various sizes, we propose a constrained random walker technique, including an adaptive matching algorithm to extract the local regions of each candidate adaptively based on the constraint of size and shape. The experimental results demonstrate that the proposed method can accurately detect multi-scale small targets in complex scenes, resulting in better detection performance than the state-of-the-art methods.

This paper introduces a novel technique for the automatic identification of real world objects in complex scenes. The identification problem requires the comparison of assemblies of image regions with a previously stored view of a known prototype. Shape context representation and matching are employed for recovering point correspondences between the image and the prototype object. This work is inspired from the problem that, a number of ambient conditions such as partial object occlusion and contour distortion, may affect the performance of the matching process and consequently the identification result. In our approach, subtractive clustering is applied in a novel fashion to enable the identification of regions of interest on the complex scene, based on a density metric. In order to increase the robustness of the identifier to mismatches and reduce the computational cost of the process, a selection of the initial suspicious regions is performed. The performance of the identifier has been examined in a great range of image and prototype selections.

Salient object detection in complex scenes and environments is a challenging research topic. Most works focus on RGB-based salient object detection, which limits its performance of real-life applications when confronted with adverse conditions such as dark environments and complex backgrounds. Taking advantage of RGB and thermal infrared(RGBT) images becomes a new research direction for detecting salient objects in complex scenes, since the thermal infrared spectrum provides the complementary information and has been used in many computer vision tasks. However, current research for RGBT salient object detection is limited by the lack of a large-scale dataset and comprehensive benchmark. This work contributes such a RGBT image dataset named VT5000, including 5000 spatially aligned RGBT image pairs with ground truth annotations. VT5000 has 11 challenges collected in different scenes and environments for exploring the robustness of algorithms. With this dataset, we propose a powerful baseline approach, which extracts multilevel features of each modality and aggregates these features of all modalities with the attention mechanism for accurate RGBT salient object detection. To further solve the problem of blur boundaries of salient objects, we also use an edge loss to refine the boundaries. Extensive experiments show that the proposed baseline approach outperforms the state-of-the-art methods on VT5000 dataset and other two public datasets. In addition, we carry out a comprehensive analysis of different algorithms of RGBT salient object detection on VT5000 dataset, and then make several valuable conclusions and provide some potential research directions for RGBT salient object detection. Our new VT5000 dataset is made publicly available at https://github.com/lz118/RGBT-Salient-Object-Detection.

We are developing a helper robot that carries out tasks ordered by users through speech. The robot needs a vision system to recognize objects appearing in the orders. However, conventional vision systems cannot recognize objects in complex scenes. They may find many objects and cannot determine which is the target. This paper proposes a method of using a conversation with the user to solve this problem. The robot asks a question to which the user can easily answer and whose answer can efficiently reduce the number of candidate objects. It considers the characteristics of features used for object recognition such as the easiness for humans to specify them by word, generating a user-friendly and efficient sequence of questions. Experimental results show that the robot can detect target objects by asking the questions generated by the method.

Visual exploration of unknown environments is considered a typical and highly important task in intelligent robotics. Although robots with visual capabilities comparable to human skills (e.g. mushroom-picking robots or bird-viewing robots) are apparently unachievable in the near future, but the concept of robots able to search for known objects in unknown surroundings is one of the ultimate goals for machine vision applications. In the scenarios that are currently envisaged, the expectations should be realistically limited. Nevertheless, one can expect that a robot, after a visual presentation of an object of interest, should be able to “learn” it and, subsequently, to detect the same object in complex scenes which may be degraded by typical effects, i.e. partial visibility of the objects (due to occlusions and/or poor illumination) and their unpredictable locations. The purpose of this chapter is to propose a novel mechanism that is potentially useful (it has been confirmed by promising preliminary results) in such applications. Several theories exist explaining the human perception of objects (e.g. Edelman, 1997). Some researchers promote the importance of multiple model views (e.g. Tarr et al., 1997) others (e.g. Biederman, 1987) postulate viewpoint invariants in the form of shape primitives (geons). From all the theories, however, the practical conclusion is that vision systems detecting objects in a human-like manner should use locally-perceived features as the fundamental tool for matching the scene content to the models of known objects. The idea of using local features (keypoints, local visual saliencies, interest points, characteristic points, corner points – several almost equivalent names exist) in machine vision can be traced back to the 80’s (e.g. Moravec, 1983; Harris & Stephens, 1988). Although stereovision and motion tracking were initially the most typical applications, it was later found that the same approach can be used in more challenging tasks (e.g. matching images in order to detect partially hidden objects). A well-known Harris-Plessey operator (Harris & Stephens, 1988) was combined with local descriptors of detected points to solve object recognition problems in which local features from analysed images are matched against a database of images depicting known objects (e.g. Schmid & Mohr, 1995). The intention was to retrieve images containing arbitrarily rotated and partially occluded objects. Subsequently developed keypoint detectors address the issues of scale changes (this was the weakest point of the original detectors) and perspective distortions. Generally, to achieve

Reliable and stable visual perception systems are needed for humanoid robotic assistants to perform complex grasping and manipulation tasks. The recognition of the object and its precise 6D pose are required. This paper addresses the challenge of detecting and positioning a textureless known object, by estimating its complete 6D pose in cluttered scenes. A 3D perception system is proposed in this paper, which can robustly recognize CAD models in cluttered scenes for the purpose of grasping with a mobile manipulator. Our approach uses a powerful combination of two different camera technologies, Time-Of-Flight (TOF) and RGB, to segment the scene and extract objects. Combining the depth image and gray image to recognize instances of a 3D object in the world and estimate their 3D poses. The full pose estimation process is based on depth images segmentation and an efficient shape-based matching. At first, the depth image is used to separate the supporting plane of objects from the cluttered background. Thus, cluttered backgrounds are circumvented and the search space is extremely reduced. And a hierarchical model based on the geometry information of a priori CAD model of the object is generated in the offline stage. Then using the hierarchical model we perform a shape-based matching in 2D gray images. Finally, we validate the proposed method in a number of experiments. The results show that utilizing depth and gray images together can reach the demand of a time-critical application and reduce the error rate of object recognition significantly.

Compared with RGB images, light field images contain more abundant visual information, which is helpful to accurately detect salient objects in complex scenes. However, most of the existing light field saliency detection methods use single light field data or do not fully consider the differences and complementarities between different light field data, resulting in insufficient multi-modal fusion. To address these issues, a multi-modal feature fusion network is proposed, which makes full use of the rich visual information in the light field images to realize the accurate saliency object detection. The proposed network consists of two parallel subnets, which are used to process the micro-lens image array and all-foucs image respectively. Then the light field refinement module is used to refine the feature map extracted from the micro-lens array stream, and finally the multi-modal feature fusion is realized by the light field attention module to predict saliency objects more accurately. In order to verify the effectiveness of proposed method, extensive comparison with several existing light field saliency detection algorithms is carried on both Lytro-Illum and LFSD datasets. Experimental results show that the proposed method is superior to others in all evaluation metrics on Lytro-Illum dataset, and has desired generalization abilities on LFSD dataset.

Perceptual localization and focus refinement network for RGB-D salient object detection

Open-vocabulary object detection is an innovative computer vision task capable of widely recognizing and localizing various objects in images. Unlike traditional methods, it can handle diverse object categories and is suitable for real-time applications in dynamic environments. Existing methods typically achieve zero-shot detection capabilities by fusing images and text. However, when semantic discrepancies exist between text and images, biased prediction problems arise, diminishing the effectiveness of semantic guidance. To address these issues, we propose a universal open-vocabulary object detection method that leverages foundational models to provide fine-grained semantic guidance for the detection process. We design a multi-level adaptive scene perception algorithm that captures subtle features of target objects in complex scenes, enabling precise separation of background and foreground. Additionally, we introduce the Text-KAN (T-KAN) model, which integrates textual descriptions with image features. By employing learnable activation functions, it resolves dependencies on linear matrices, enhances text interpretability, corrects semantic biases, and achieves precise alignment between images and text at a fine-grained level. We comprehensively evaluate the performance of our proposed method on existing open-vocabulary benchmarks, conducting experiments on the COCO and LVIS datasets. The results demonstrate significant performance gains in detecting novel categories, highlighting the method’s strong generalization capabilities. This work provides valuable insights and references for advancing the field of open-vocabulary object detection.

Robust detection of moving objects in complex scenes is one of the most challenging issues in computer vision. In this paper, we present a novel texture-wise approach to segment moving objects with codebook and local binary patterns (LBP). In many moving segmentation algorithms, the information from limited frames before current image is used. Our approach models background over long time with small memory. Firstly, we construct codebook model which represents a compressed form of background model for long image sequences. A single Gaussian model of per-pixel is built to deal with illumination changes. By using the correlation and texture of spatially proximal pixels, local binary patterns background model is constructed. Finally current image is segmented into two parts, foreground and background, by comparing current image with background model. Experiments show that the proposed approach achieves promising results robustly in real videos.

Hyperspectral unmixing (HU) aims to obtain subpixel-level material composition information, which is crucial for the precise advancement of hyperspectral image processing techniques. In recent years, deep learning (DL) has been widely applied to HU due to its strong ability to capture complex and nonlinear feature relationships in the data. However, relying solely on hyperspectral images for unmixing often fails to effectively distinguish objects in complex scenes, particularly when different endmembers exhibit similar spectral characteristics. To address this limitation, we propose a Multimodal Fusion Network (MMFNet) that incorporates the elevation information inherent in light detection and ranging (LiDAR) data. MMFNet is capable of simultaneously extracting spectral features from hyperspectral images and spatial features from LiDAR data. Furthermore, most existing DL-based HU methods operate only in the original spectral domain, which makes them susceptible to spectral variability, noise, and limited discriminative capacity. To overcome these challenges, we integrate Learnable Wavelet Transform (LWT) into MMFNet to adaptively decompose hyperspectral signals into multiple frequency subdomains, thereby mitigating spectral variability and noise while preserving spatial consistency. In addition, a Multi-Scale Convolution Fusion Module (MSCFM) is designed to capture semantic information at different receptive fields and enhance the fine-grained fusion of spectral and spatial features. Through these designs, MMFNet produces more robust and discriminative feature representations, enabling better separation of spectrally similar endmembers. Extensive experiments demonstrate that the proposed MMFNet outperforms several state-of-the-art unmixing methods.