MDF-SLAM: Monocular Dense 3D Reconstruction Based on Depth Estimation

Depth estimation and camera calibration of a focused plenoptic camera for visual odometry

This article presents a new sensor fusion method for visual simultaneous localization and mapping (SLAM) through integration of a monocular camera and a 1D-laser range finder. Such as a fusion method provides the scale estimation and drift correction and it is not limited by volume, e.g., the stereo camera is constrained by the baseline and overcomes the limited depth range problem associated with SLAM for RGBD cameras. We first present the analytical feasibility for estimating the absolute scale through the fusion of 1D distance information and image information. Next, the analytical derivation of the laser-vision fusion is described in detail based on the local dense reconstruction of the image sequences. We also correct the scale drift of the monocular SLAM using the laser distance information which is independent of the drift error. Finally, application of this approach to both indoor and outdoor scenes is verified by the Technical University of Munich dataset of RGBD and self-collected data. We compare the effects of the scale estimation and drift correction of the proposed method with the SLAM for a monocular camera and a RGBD camera.

This paper presents an object detection method that can simultaneously estimate the positions and depth of the objects from multiplexed images. Multiplexed image is produced by a new type of imaging device that collects the light from different fields of view using a single image sensor, which is originally designed for stereo, 3D reconstruction and broad view generation using computational imaging. Intuitively, multiplexed image is a blended result of the images of multiple views and both of the appearance and disparities of objects are encoded in a single image implicitly, which provides the possibility for reliable object detection and depth/disparity estimation. Motivated by the recent success of CNN based detector, a multi-anchor detector method is proposed, which detects all the views of the same object as a clique and uses the disparity of different views to estimate the depth of the object. The proposed method is interesting in the following aspects: firstly, both locations and depth of the objects can be simultaneously estimated from a single multiplexed image; secondly, there is almost no computation load increase comparing with the popular object detectors; thirdly, even in the blended multiplexed images, the detection and depth estimation results are very competitive. There is no public multiplexed image dataset yet, therefore the evaluation is based on the simulated multiplexed image using the stereo images from KITTI, and very encouraging results have been obtained.

Light-field (LF) camera holds great promise for passive/general depth estimation benefited from high angular resolution, yet suffering small baseline for distanced region. While stereo solution with large baseline is superior to handle distant scenarios, the problem of limited angular resolution becomes bothering for near objects. Aiming for all-in-depth solution, we propose a cross-baseline LF camera using a commercial LF camera and a monocular camera, which naturally form a 'stereo camera' enabling compensated baseline for LF camera. The idea is simple yet non-trivial, due to the significant angular resolution gap and baseline gap between LF and stereo cameras.

In this paper, we propose an online scene reconstruction algorithm with monocular camera since there are many advantages on modeling and visualization of an environment with physical scene reconstruction instead of resorting to sparse 3D points. The goal of this algorithm is to simultaneously track the camera position and map the 3D environment, which is close to the spirit of visual SLAM. There're plenty of visual SLAM algorithms in the current literature which can provide a high accuracy performance, but many of them rely on stereo cameras. It's true that we'll face many more challenges to accomplish this task with monocular camera. However, the advantages of cheaper and easier deployable hardware setting have made monocular approach more attractive. Specifically, we apply a maximum a posteriori Bayesian approach with optimization technique to simultaneously track the camera and build a dense point cloud. We also propose a feature expansion method to expand the density of points, and then online reconstruct the scene with a delayed approach. Furthermore, we utilize the reconstructed model to accomplish visual localization task without extracting the features. Finally, a number of experiments have been conducted to validate our proposed approach, and promising performance can be observed.

The rapid spreading of RGB-D cameras has led to wide applications of 3D videos in both academia and industry, such as 3D entertainment and 3D visual understanding. Under these circumstances, extensive research efforts have been dedicated to RGB-D camera--oriented topics. In these topics, quality promotion of depth videos with the temporal characteristic is emerging and important. Due to the limited exposure time of RGB-D cameras, object movement can easily lead to motion blurs in intensive images, which can further result in obvious artifacts (holes or fake boundaries) in the corresponding depth frames. With regard to this problem, we propose a depth error elimination method based on time series analysis to remove the artifacts in depth images. In this method, we first locate the regions with erroneous depths in intensive images by using motion blur detection based on a time series analysis model. This is based on the fact that the depth image is calculated by intensive color images that are captured synchronously by RGB-D cameras. Then, the artifacts, such as holes or fake boundaries, are fixed by a depth error elimination method. To evaluate the performance of the proposed method, we conducted experiments on 250 images. Experimental results demonstrate that the proposed method can locate the error regions correctly and eliminate these artifacts effectively. The quality of depth video can be improved significantly by using the proposed method.

Results of using an RGB-D camera (Kinect sensor) and a stereo camera, separately, in order to determine the 3D real position of characteristic points of a predetermined object in a scene are presented. KAZE algorithm was used to make the recognition, that algorithm exploits the nonlinear scale space through nonlinear diffusion filtering; 3D coordinates of the centroid of a predetermined object were calculated employing the camera calibration information and the depth parameter provided by a Kinect sensor and a stereo camera. Experimental results show it is possible to get the required coordinates with both cameras in order to locate a robot, although a balance in the distance where the sensor is placed must be guaranteed: no fewer than 0.8 m from the object to guarantee the real depth information, it is due to Kinect operating range; 0.5 m to stereo camera, but it must not be 1 m away to have a suitable rate of object recognition, besides, Kinect sensor has more precision with distance measures regarding a stereo camera.

Urban mapping for street-side environments (i.e., sidewalk environments rather than auto-centric ones) is just one of many new applications requiring simultaneous object detection and localization in outdoor scenes. Automated mapping has focused on car environments in support of vehicle travel, though pedestrian infrastructure maps, historical structure maps and recreational trail maps are as important in supporting urban life. Unfortunately, many of the auto-centric automated approaches cannot be leveraged in other outdoor use cases due to challenging and cost-prohibitive depth sensing hardware involving LiDar sensors and large computing resources. Using depth-sensing stereo cameras is cost-efficient, but little information is available detailing the best uses and efficacy of different systems in diverse outdoor environments. Our review highlights the complexity of localizing objects in real outdoor environments using stereo cameras and our comparative analysis approach to two common camera setups demonstrates the variation among depth-sensing cameras. We collect a modest dataset with side-by-side images for comparing two commonly used depth cameras, the ZED Stereo Camera (a passive stereo camera) and Intel® RealSense™Depth Camera D435 (an active IR stereo camera). We contrast system performance in diverse urban environments, and provide accuracy of depth measurement results based on data collected from each stereo sensor. We focus our discussion on practical considerations and objects that are commonly presented in the street environment but usually hard to map by hand.

We propose simple three-dimensional measurement system in this paper. We think this system introduce a vehicle. This system is cheaper and easier than existing methods because, this system need only monocular camera and one light source. In addition we propose method re-create absorbed luminance by the texture to be able to measure any objects. Proposed method must keep pace surface that have high absorptive power because, proposed method use luminance distribution. This method use two light source. It can introduce vehicle very easy because, vehicle have back light and tail lamp.

In numerous applications, such as collision detection, Robotic handling, Robotic-based manufacturing facility, and Advanced Driver Assistance Systems (ADAS), depth estimation is crucial and the most significant task. Radar, Ultrasonic, Lidar technologies both operate by reflecting radio or sound wave or laser beams respectively. Stereo cameras used for depth estimation are costly and increase the cost of the system. There are a few Monocular camera depth estimate approaches that have evolved using mathematical calculation. One approach uses pixel to depth estimate mapping and the other uses the geometry of the road, the contact point of the vehicle on road, and camera properties. The proposed solution would implement depth detection using a monocular camera with deep learning. The objective of the proposed solution is to detect the depth from a monocular image and calibrate it with the actual depth. The pixel to distance data which derived from camera properties is used to run through varied hidden layers and nodes to conclude the implementation using Deep Learning (DL). Evidently, train and test data learn and converge more quickly on the deeper models which has more hidden layers and nodes.

Although unsupervised methods of monocular depth and camera motion estimation have made significant progress, most of them are based on the static scene assumption and may perform poorly in dynamic scenes. In this paper, we propose a novel framework for unsupervised learning of monocular depth and camera motion estimation, which is applicable to dynamic scenes. Firstly, the framework is trained to obtain initial inference results by assuming the scene is static, through minimizing a photometric consistency loss and a 3D transformation consistency loss. Then, the framework is fine-tuned by jointly learning with a motion rectification network (RecNet). Specifically, RecNet is designed to rectify the individual motion of moving objects and generate motion rectified images, enabling the framework to learn accurately in dynamic scenes. Extensive experiments have been done on the KITTI dataset. Results show that our method achieves state-of-the-art performance on both depth prediction and camera motion estimation tasks.

We propose a range estimation method for vision-based forward collision warning systems with a monocular camera. To solve the problem of variation of camera pitch angle due to vehicle motion and road inclination, the proposed method estimates virtual horizon from size and position of vehicles in captured image at run-time. The proposed method provides robust results even when road inclination varies continuously on hilly roads or lane markings are not seen on crowded roads. For experiments, a vision-based forward collision warning system has been implemented and the proposed method is evaluated with video clips recorded in highway and urban traffic environments. Virtual horizons estimated by the proposed method are compared with horizons manually identified, and estimated ranges are compared with measured ranges. Experimental results confirm that the proposed method provides robust results both in highway and in urban traffic environments.

This paper reports on an algorithm for automatic, targetless, extrinsic calibration of a lidar and optical camera system based upon the maximization of mutual information between the sensor‐measured surface intensities. The proposed method is completely data‐driven and does not require any fiducial calibration targets—makingin situcalibration easy. We calculate the Cramér‐Rao lower bound (CRLB) of the estimated calibration parameter variance, and we show experimentally that the sample variance of the estimated parameters empirically approaches the CRLB when the amount of data used for calibration is sufficiently large. Furthermore, we compare the calibration results to independent ground‐truth (where available) and observe that the mean error empirically approaches zero as the amount of data used for calibration is increased, thereby suggesting that the proposed estimator is a minimum variance unbiased estimate of the calibration parameters. Experimental results are presented for three different lidar‐camera systems: (i) a three‐dimensional (3D) lidar and omnidirectional camera, (ii) a 3D time‐of‐flight sensor and monocular camera, and (iii) a 2D lidar and monocular camera.

In-water visual ship hull inspection using unmanned underwater vehicles needs to be performed at very close range to the target surface because of the visibility limitations in underwater environments mainly due to light attenuation, scattering, and water turbidity. These environmental challenges result in ineffective photometric and geometric information in hull surface images and, therefore, the performance of conventional three-dimensional (3D) reconstruction techniques is often unsatisfactory. This paper addresses a visual mapping method for 3D reconstruction of underwater ship hull surface using a monocular camera as a primary mapping sensor. The main idea of the proposed approach is to model the moderately curved hull surface as a combination of piecewise-planar panels, and to generate a global map by aligning the local images in a two-dimensional reference frame and correcting them appropriately to reflect the information of perspective projections of the 3D panels. The estimated 3D panels associated with the local images are used to extract the loop-closure relative measurements in the framework of simultaneous localization and mapping (SLAM) for precise camera trajectory estimation and 3D reconstruction results. The validity and practical feasibility of the proposed method are demonstrated using a dataset obtained in a field experiment with a full-scale ship in a real sea environment.

Purpose This paper aims to propose a measurement principle and a calibration method of measurement system integrated with serial robot and 3D camera to identify its parameters conveniently and achieve high measurement accuracy. Design/methodology/approach A stiffness and kinematic measurement principle of the integrated system is proposed, which considers the influence of robot weight and load weight on measurement accuracy. Then an error model is derived based on the principle that the coordinate of sphere center is invariant, which can simultaneously identify the parameters of joint stiffness, kinematic and hand-eye relationship. Further, considering the errors of the parameters to be calibrated and the measurement error of 3D camera, a method to generate calibration observation data is proposed to validate both calibration accuracy and parameter identification accuracy of calibration method. Findings Comparative simulations and experiments of conventional kinematic calibration method and the stiffness and kinematic calibration method proposed in this paper are conducted. The results of the simulations show that the proposed method is more accurate, and the identified values of angle parameters in modified Denavit and Hartenberg model are closer to their real values. Compared with the conventional calibration method in experiments, the proposed method decreases the maximum and mean errors by 19.9% and 13.4%, respectively. Originality/value A new measurement principle and a novel calibration method are proposed. The proposed method can simultaneously identify joint stiffness, kinematic and hand-eye parameters and obtain not only higher measurement accuracy but also higher parameter identification accuracy, which is suitable for on-site calibration.