カメラ位置姿勢推定のためのRGB-Dカメラによる3次元線分モデル生成法

Real-time camera tracking for marker-less and unprepared augmented reality environments

This paper has two contributions in the context of line based camera pose estimation, 1) We propose a purely geometric approach to establish correspondence between 3D line segments in a given model and 2D line segments detected in an image, 2) We eliminate a degenerate case due to the type of rotation representation in arguably the best line based pose estimation method currently available. For establishing line correspondences we perform exhaustive search on the space of camera pose values till we obtain a pose (position and rotation) which is geometrically consistent with the given set of 2D, 3D lines. For this highly complex search we design a strategy which performs precomputations on the 3D model using separate set of constraints on position and rotation values. During runtime, the set of different rotation values are ranked independently and combined with each position values in the order of their ranking. Then successive geometric constraints which are much simpler when compared to computing reprojection error are used to eliminate incorrect pose values. We show that the ranking of rotation values reduces the number of trials needed by a huge factor and the simple geometric constraints avoid the need for computing the reprojection error in most cases. Though the execution time for the current MATLAB implementation is far from real time requirement, our method can be accelerated significantly by exploiting simplicity and parallelizability of the operations we employ. For eliminating the degenerate case in the state of art pose estimation method, we reformulate the rotation representation. We use unit quaternions instead of CGR parameters used by the method.

PurposeThis paper aims to propose a real-time augmented reality (AR)-based assembly assistance system using a coarse-to-fine marker-less tracking strategy. The system automatically adapts to tracking requirement when the topological structure of the assembly changes after each assembly step.Design/methodology/approachThe prototype system’s process can be divided into two stages: the offline preparation stage and online execution stage. In the offline preparation stage, planning results (assembly sequence, parts position, rotation, etc.) and image features [gradient and oriented FAST and rotated BRIEF (ORB)features] are extracted automatically from the assembly planning process. In the online execution stage, too, image features are extracted and matched with those generated offline to compute the camera pose, and planning results stored in XML files are parsed to generate the assembly instructions for manipulators. In the prototype system, the working range of template matching algorithm, LINE-MOD, is first extended by using depth information; then, a fast and robust marker-less tracker that combines the modified LINE-MOD algorithm and ORB tracker is designed to update the camera pose continuously. Furthermore, to track the camera pose stably, a tracking strategy according to the characteristic of assembly is presented herein.FindingsThe tracking accuracy and time of the proposed marker-less tracking approach were evaluated, and the results showed that the tracking method could run at 30 fps and the position and pose tracking accuracy was slightly superior to ARToolKit.Originality/valueThe main contributions of this work are as follows: First, the authors present a coarse-to-fine marker-less tracking method that uses modified state-of-the-art template matching algorithm, LINE-MOD, to find the coarse camera pose. Then, a feature point tracker ORB is activated to calculate the accurate camera pose. The whole tracking pipeline needs, on average, 24.35 ms for each frame, which can satisfy the real-time requirement for AR assembly. On basis of this algorithm, the authors present a generic tracking strategy according to the characteristics of the assembly and develop a generic AR-based assembly assistance platform. Second, the authors present a feature point mismatch-eliminating rule based on the orientation vector. By obtaining stable matching feature points, the proposed system can achieve accurate tracking results. The evaluation of the camera position and pose tracking accuracy result show that the study’s method is slightly superior to ARToolKit markers.

The automatic reconstruction of photogrammetric mesh models often suffers from issues such as erroneous adhesion of different planes, local distortions, and the loss of edge details. This study aims to improve mesh quality by incorporating three-dimensional (3D) line segments and plane regions as geometric constraints during mesh refinement. Using previously obtained 3D line segments, depth, and normal information, accurate 3D plane regions are extracted to complete missing sections of the dense point cloud, with depth-based outlier removal applied. Correspondences between mesh model vertices and the 3D line segments and plane regions are established, followed by mesh refinement through an energy function that integrates photo-consistency, line and plane constraints, and regularization. Experiments conducted on four open-source datasets demonstrate that the proposed method substantially improves 3D plane region extraction and point cloud completion. The refined mesh model achieves a 24.30% reduction in the average distance related to 3D lines and a 15.75% reduction for 3D planes compared with the original model, confirming the effectiveness of incorporating 3D line and plane constraints in enhancing reconstruction accuracy.

This paper presents a novel approach to both omnidirectional camera calibration and 3D reconstruction of the surrounding scene by contour matching in architectural scenes. By using a quantitative measure to consider the inlier distribution, we can estimate more precise camera model parameters and structure from motion. Since most of line segments of man-made objects are projected to the contours in omnidirectional images, contour matching problem is important in camera recovery process. We propose a novel 3D reconstruction method by contour matching in three omnidirectional views. First, two points on the contour and their viewing vectors are used to determine an interpretation plane equation, and we obtain a contour intersecting both the plane and the estimated patch of the camera model. Then, 3D line segment is calculated from two patches, which is projected to the contour on the third views, and these matching results are used in refinement of camera recovery.

A version of the geometrical shortest path problem is to compute a shortest path connecting two points and passing a finite set of line segments in three dimensions. This problem arises in the pursuit path problem and also be used as a tool to finding shortest paths on polyhedral surface. This paper presents an iterative algorithm for dealing with the problem, particularly with large data. The idea is to simultaneously determines on each segment a point such that the length of the path successively connecting the points is decreased. We show that after a finite number of iterations, the algorithm converges to give an approximate solution. The algorithm is implemented in C++ and tested for large datasets. The numerical results are shown and discussed.

We propose an approach to predict accuracy for three-dimensional reconstruction and camera pose using a generic RGB-D camera on a robotic platform. We initially create a ground truth of 3D points and camera poses using a set of smart markers that we specifically devised and constructed for our approach. Then, we compute actual errors and their accuracy during the motion of our mobile robotic platform. The error modeling is then provided, which is used as input to a deep multi-layer perceptron in order to estimate accuracy as a function of the camera’s distance, velocity, and vibration of the vision system. The network outputs are the root mean squared errors for the 3D reconstruction and the relative pose errors for the camera. Experimental results show that this approach has a prediction accuracy of ±1% for the 3D reconstruction and ±2.5% for camera poses, which shows a better performance in comparison with state-of-the-art methods.

In recent years, low-cost and lightweight RGB and depth (RGB-D) sensors, such as Microsoft Kinect, have made available rich image and depth data, making them very popular in the field of simultaneous localization and mapping (SLAM), which has been increasingly used in robotics, self-driving vehicles, and augmented reality. The RGB-D SLAM constructs 3D environmental models of natural landscapes while simultaneously estimating camera poses. However, in highly variable illumination and motion blur environments, long-distance tracking can result in large cumulative errors and scale shifts. To address this problem in actual applications, in this study, we propose a novel multithreaded RGB-D SLAM framework that incorporates a highly accurate prior terrestrial Light Detection and Ranging (LiDAR) point cloud, which can mitigate cumulative errors and improve the system’s robustness in large-scale and challenging scenarios. First, we employed deep learning to achieve system automatic initialization and motion recovery when tracking is lost. Next, we used terrestrial LiDAR point cloud to obtain prior data of the landscape, and then we applied the point-to-surface inductively coupled plasma (ICP) iterative algorithm to realize accurate camera pose control from the previously obtained LiDAR point cloud data, and finally expanded its control range in the local map construction. Furthermore, an innovative double window segment-based map optimization method is proposed to ensure consistency, better real-time performance, and high accuracy of map construction. The proposed method was tested for long-distance tracking and closed-loop in two different large indoor scenarios. The experimental results indicated that the standard deviation of the 3D map construction is 10 cm in a mapping distance of 100 m, compared with the LiDAR ground truth. Further, the relative cumulative error of the camera in closed-loop experiments is 0.09%, which is twice less than that of the typical SLAM algorithm (3.4%). Therefore, the proposed method was demonstrated to be more robust than the ORB-SLAM2 algorithm in complex indoor environments.

Line segment based axial motion stereo

In an autonomous driving environment, the interactive platform technology inside the vehicle is important. To this end, interactive technology that allows multiple users to interact naturally in real-time through transparent display devices and technology that combines external real-world and augmented reality based on multiple scenarios have been developed. In such a fusion technology, position/orientation information of a vehicle for sensing an outdoor environment is important. In this paper, we propose camera pose estimation using voxel-based features for vehicle position and orientation tracking. Our method is a camera pose estimation method based on a 2D-3D matching scheme using multi-task learning that utilize CNNs(Convolutional Neural Networks) for performance improvement. The scene is divided into voxels, and the 3D points of each voxel are projected back onto the image using ground truth camera pose, and the polygonal area is identified using convex hull. We designed a training dataset for 2D-3D matching, which consists of internal 3D points, correspondence between pairs of images, and voxel regions in the image. We train the learning structure with three multi-task CNNs to obtain voxel region detection and the location and description of the points of interest in it. Thereafter, a 2D-3D matching correspondence is formed, and based on this, the camera pose is estimated by solving the PnP(Perspective-n-Point) problem.

In this paper, we propose a novel moving-objects detection method which, in contrast to state-of-the-art moving-objects detection methods, takes static feature points into consideration during detection. It benefits both the tracking and mapping approaches in real-time Simultaneous Localization and Mapping (SLAM) system whose localization depends on static objects primarily. Our method obtains accurate static feature point sets continuously to estimate camera poses. In return, these camera poses are employed to find inliers and outliers using succinct distinction method. More specifically, we estimate camera pose by using static map points extracted from a high accurate global map. The map is generated in a real-time stereo SLAM system, and the camera pose is estimated and optimized by using a 3D-2D projection matching search with local bundle adjustment optimization. Once we have a calculated camera pose, we begin to find inliers and outliers. Finally, combined with superpixel segmentation, we capture the moving objects and then give feedback to the whole SLAM system. Detailed results are demonstrated comparing to other moving-objects detection methods on selected KITTI datasets with moving objects.

Unsupervised cycle optimization learning for single-view depth and camera pose with Kalman filter

We propose a hybrid camera pose estimation method using an inclination sensor value and correspondence-free line segments. In this method, possible azimuths of the camera pose are hypothesized by a voting method under an inclination constraint. Then some camera positions for each possible azimuth are calculated based on the detected line segments that affirmatively voted for the azimuth. Finally, the most consistent one is selected as the camera pose out of the multiple sets of the camera positions and azimuths. Unlike many other tracking methods, our method does not use past information but rather estimates the camera pose using only present information. This feature is useful for an initialization measure of registration in augmented reality (AR) systems. This paper describes the details of the method and shows its effectiveness with experiments in which the method is actually used in an AR application.

This paper reports a new methodology for an automatic extraction of vertices coordinates from a dental wire CAD model in an initial graphics exchange specification (IGES) file format for CNC bending code generation. It has two main components, the first one deals with IGES feature extraction and the second one deals with autonomous CNC code generation based on the Cartesian coordinates using various mathematical formulae. The procedure is carried out in MATLAB environment through some developed functions to support wire design technique in CAD by means of 3D line segmentation. In this approach, the digital design of the wire is constructed from multiple 3D line segments in reference to the 3D teeth scan image (STL file). The developed methodology has three steps consisting of an IGES model pre-processing, automated coordinate extraction and CNC bending code generation. The developed methodology is limited to the CAD design through multiple line segments and has been verified with a case study.

A MPB-based remote sensing image 3D reconstruction system