T-Net: Effective Permutation-Equivariant Network for Two-View Correspondence Learning

PESA-Net: Permutation-Equivariant Split Attention Network for correspondence learning

Due to the rapid development of mobile devices, augmented reality (AR) has been utilized increasingly, and thus applications of AR have increased for indoor and outdoor scenes. However, most of them focus on dealing with indoor scenes and the research for outdoor scenes is rare. In this paper, we propose a system to explore an application of AR for presenting 3D architecture models in outdoor scenes. We aim to enhance the realism of a 3D architectural model for an outdoor scene and provide users with a realistic experience. We first capture a color image through a camera and detect a plane on the image. When obtaining the detected plane, we can generate an anchor and attach virtual objects (3D architecture models) to this anchor. After that, we use the environmental high dynamic range mode of ARCore to obtain lighting information. Thus, our system can present a more realistic light and dark contrast and shadows on the virtual objects. Moreover, we propose an occlusion handling technique to obtain the correct relative positions of real objects and virtual objects. We use a depth prediction model to obtain depth information enabling the real objects to occlude the virtual objects. Finally, the processed augmented color image is rendered. The experiments demonstrate that our system can produce favorable results within a reasonable distance.

The existing 3D point cloud classification/segmentation networks directly use Convolutional Neural Networks (C-NNs) to extract features from indoor data and have no advantages handling complex outdoor scenes. This is mainly due to the segmentation of large scale outdoor scenes depends on global context information. Inspired by Global Attention (GA) mechanism, we design a Global Point Attention module (GPA) by regarding high-level features, which usually contain rich semantic information, as a guidance to low-level features. In this paper, we embed GPA in PointSIFT to accomplish segmentation and call this new network PointSIFT-GPA. Experimental results on the US3D dataset demonstrate the performance of GPA and the superior performance of PointSIFT-GPA. In particular, PointSIFT-GPA ranks the 2-nd place on 2019 IEEE GRSS Data Fusion Contest 3D Point Cloud Classification Challenge with mIoU 0.9454.

Slam has important applications in the field of unmanned systems and robots. At present, most slam algorithms have satisfied results in indoor scenes, but the pose drift will occur in outdoor complex scenes, and most 3D SLAM can not generate effective navigation maps. Firstly, we introduce the basic method. Secondly, we propose the enhanced pretreatment and mapping methods based on lightweight lidar odometry, including a map dynamic update method based on double-layer boundary, which can generate 2D map without dynamic noise. And the ground point cloud segmentation method based on straight line fitting with ground points, which can reduce the pose drift. Thirdly, experiment results show that the SLAM algorithm proposed in this paper effectively solves the drift problem in the mapping of outdoor complex scenes and generates a navigable 2D-3D Hybrid map.

This paper presents an efficient method to achieve real-time drift-free path replaying for Micro Aerial Vehicle (MAV). The drift-free path replaying method is proposed for precise autonomous path following in GPS-denied environments. In this paper, we present solutions on how to effectively select recording images during flying and drive the MAV to fly automatically to target position. During the path replaying, a strategy named triggered virtual image matching is used for reducing the computational cost and eliminating the drift. Time of image selection is reduced to negligible due to above strategy. Moreover, we use optical flow method for features matching between current image and the virtual image which is precise and faster. Experimental results show that our approach can achieve autonomous returning and precise path following in indoor scene and outdoor scene.

Solid-state light detection and ranging (LiDAR) has the advantages of low cost, small size and strong practicability. However, it faces challenges in simultaneous localization and mapping applications due to its small field of view and irregular scanning patterns. A solid-state LiDAR simultaneous localization and mapping system containing intensity information is proposed. In order to solve the irregular scanning characteristics of solid-state LiDAR, we introduce a data preprocessing framework and add intensity feature points to the front-end odometer. This improves the accuracy and robustness of positioning when geometric feature points are scarce, thus solving the problem of feature point degradation caused by a finite field of view. In the back-end optimization stage, we combine the geometric feature residuals with the intensity feature residuals through the proposed consistent difference function, so that the system can maintain good performance even in challenging environments. Finally, we conducted an extensive evaluation of the proposed algorithm on official datasets and various datasets collected from multiple platforms, and the results confirmed the validity of our approach. Compared with the corresponding method, in indoor scenes, the absolute trajectory error and relative attitude error are decreased by 54.5% and 5.3%. In outdoor scenes, the absolute trajectory error and relative attitude error are decreased by 29.6% and 58.8%.

Inter-frame matching of laser point clouds is the key to SLAM's autonomous positioning and 3D map construction. Aiming at the failure of inter frame matching of multi-line lidar SLAM during fast field of view conversion, this paper proposes a spatial 3D mapping method using 180 degree large-field-angle lidar to improve A-LOAM. Firstly, the nonground feature points are obtained by dual-threshold ground filtering, and a novel point cloud feature extraction method based on curvature is used to realize the feature point matching of large-field-angle point cloud; Secondly, using the matching methods of edge point to edge line and plane point to plane block, through the robust frame matching of large-field-angle point cloud data, the spatial 3D mapping is realized; Finally, three-dimensional mapping experiments are carried out on the large-field-angle point cloud data collected in different scenes. The results show that the spatial 3D mapping based on robust frame matching of large-field-angle lidar can reach the accuracy of nearly centimeter level in indoor scene and decimeter level in outdoor scene. It has application prospects in 3D mapping and other fields.

Compared to fully supervised 3D large-scale point cloud segmentation methods, which necessitate extensive manual point-wise annotations, weakly supervised segmentation has emerged as a popular approach for significantly reducing labeling costs while maintaining effectiveness. However, the existing methods have exhibited inferior segmentation performance and unsatisfactory generalization capabilities in some scenarios with unique structures (e.g., building facades). In this paper, we propose an effective and generalized weakly supervised semantic segmentation framework, called multi-stage scene-level constraints (MSC), to solve the above problem. To address the issue regarding inadequate labeled data, we use pseudo-labels for unlabeled data and propose an uncertainty-guided adaptive reweighting strategy to reduce the negative impact of erroneous pseudo-labeled data on the model learning process. To address the class imbalance issue, we employ multi-stage scene-level constraints (i.e., encoder, decoder, and classifier stages) to treat each class equally and improve perception ability of the model for each class. Evaluations conducted on multiple large-scale point cloud datasets collected in different scenarios, including building facades, indoor scenes, outdoor scenes, and UAV scenes, show that our MSC achieves a large gain over the existing weakly supervised methods and even surpasses some fully supervised methods.

The image super-resolution (SR) technique, which aims at reconstructing a high-resolution (HR) image from a single low-resolution (LR) image, is a classical problem in computer vision. Limited by the imaging hardware, the spatial resolution of a hyperspectral images (HSI) is usually very coarse. Meanwhile, the spectral information of the HSI is extremely important for its applications and cannot be severely distorted. This paper presents a spatial constraint (SCT) strategy with combination of a deep learning method for HSI SR. The SCT strategy restraints the LR HSI generated by the reconstructed HR HSI should be spatially close to the input LR HSI. The deep learning method learns an end-to-end mapping between the spectral difference of the LR HSI and that of the HR HSI. The mapping is represented as a deep convolutional neural network (CNN). The CNN learned spectral difference is utilized to super-resolve the LR HSI while preserve the important spectral information of the desired HR HSI. Experiments have been conducted on three databases that contains both indoor scenes and outdoor scenes. Comparative analyses have verified the effectiveness of the overall method.

Establishing reliable correspondences from a putative correspondence set is a challenging task. Most of state-of-the-art methods utilize the local context and global context to address the task. However, the local and global context often contains large number of outliers, which have a negative impact on capturing scene geometry. In this paper, we propose a Multi-scale Attention Network (called MANet), which introduces the attention mechanism for feature matching, to improve the ability of capturing scene geometry. Specifically, we first fuse the features of low and high levels by an multi-scale strategy network to enhance the representative ability of features. Then, we propose an attentive PointCN block and an attentive pooling layer, to discriminatively capture global context and local context information, respectively. We demonstrate through extensive experiments on both indoor and outdoor datasets that MANet provides a significant improvement in the performance of the two-view geometry and correspondences accuracy compared to the state-of-the-art methods.

In the field of remote sensing, cloud cover severely reduces the quality of satellite observations of the earth. Due to the complete absence of information in cloud-covered regions, cloud removal with a single optical image is an ill-posed problem. Since the synthetic aperture radar (SAR) can effectively penetrate clouds, fusing SAR and optical remote sensing images will effectively alleviate this problem. However, existing SAR-based optical cloud removal methods fail to effectively leverage the global information provided by the SAR image, resulting in limited performance gains. In this paper, we introduce a novel cloud removal method named the Multi-Level SAR-Guided Contextual Attention Network (MSGCA-Net). MSGCA-Net is designed with a multi-level architecture that integrates a SAR-Guided Contextual Attention (SGCA) module to fuse the dependable global contextual information from SAR images with the local features of optical images effectively. In the module of SGCA, the SAR image provides reliable global contextual information and genuine structure of cloud-covered regions, while the optical image provides the local feature information. The proposed model can efficiently extract and fuse global and local contextual information in SAR and optical images. We trained and evaluated the performance of the model on both simulated and real-world datasets. Both qualitative and quantitative experimental evaluation demonstrated that the proposed method can yield high quality cloud-free images and outperform state-of-the-art cloud removal methods.

The ray projection and its application to recovering a projective geometric transformation and an affine lighting change between two objects are mathematically studied. A novel technique, viz., variable contour, is proposed for accurately evaluating the ray projection. Moreover, a novel flexible framework of the ray projection is devised for the joint recovery of the eight parameters for the projective transformation and the two parameters for the lighting change. Finally, the framework is experimentally evaluated by recovering a variety of geometric and photometric transformations between real images of indoor and outdoor scenes. Its robustness to image blur and occlusion is demonstrated. Its versatility is illustrated through matching different objects in different classes.

We present a method for rendering novel views from a set of reference views under the assumption that scene surfaces can be approximated by planar patches. We use a set of sparse correspondences between the reference views to determine homographies through different planar patches using a clustering technique. We also obtain a segmentation of the scene in terms of planar regions visible from at least two views. Such segmentations explicitly resolve the visibility issues in novel view generation. We present results on rendering of realistic walk-through sequences for both indoor and outdoor scenes which demonstrate the applicability of our method.

In the context of improved navigation for micro aerial vehicles, a new scene recognition visual descriptor, called spatial color gist wavelet descriptor (SCGWD), is proposed. SCGWD was developed by combining proposed Ohta color-GIST wavelet descriptors with census transform histogram (CENTRIST) spatial pyramid representation descriptors for categorizing indoor versus outdoor scenes. A binary and multiclass support vector machine (SVM) classifier with linear and non-linear kernels was used to classify indoor versus outdoor scenes and indoor scenes, respectively. In this paper, we have also discussed the feature extraction methodology of several, state-of-the-art visual descriptors, and four proposed visual descriptors (Ohta color-GIST descriptors, Ohta color-GIST wavelet descriptors, enhanced Ohta color histogram descriptors, and SCGWDs), in terms of experimental perspectives. The proposed enhanced Ohta color histogram descriptors, Ohta color-GIST descriptors, Ohta color-GIST wavelet descriptors, SCGWD, and state-of-the-art visual descriptors were evaluated, using the Indian Institute of Technology Madras Scene Classification Image Database two, an Indoor-Outdoor Dataset, and the Massachusetts Institute of Technology indoor scene classification dataset [(MIT)-67]. Experimental results showed that the indoor versus outdoor scene recognition algorithm, employing SVM with SCGWDs, produced the highest classification rates (CRs)—95.48% and 99.82% using radial basis function kernel (RBF) kernel and 95.29% and 99.45% using linear kernel for the IITM SCID2 and Indoor-Outdoor datasets, respectively. The lowest CRs—2.08% and 4.92%, respectively—were obtained when RBF and linear kernels were used with the MIT-67 dataset. In addition, higher CRs, precision, recall, and area under the receiver operating characteristic curve values were obtained for the proposed SCGWDs, in comparison with state-of-the-art visual descriptors.

As contemporary people’s pursuit of spiritual quality becomes more and more personalized, the traditional two-dimensional design concept cannot meet the needs of modern people. Based on the neural network, this study combined intelligent simulation technology to simulate the home environment and carried out the corresponding interior design. The system simulated indoor and outdoor scenes by combining smart home and virtual reality technology. At the same time, this paper focused on simulating the demonstration control of indoor home functions and analyzed them according to the actual design situation. The research shows that the proposed algorithm has certain effects and can provide theoretical reference for related research.