Simultaneous localization and scene reconstruction with monocular camera

This article presents a new sensor fusion method for visual simultaneous localization and mapping (SLAM) through integration of a monocular camera and a 1D-laser range finder. Such as a fusion method provides the scale estimation and drift correction and it is not limited by volume, e.g., the stereo camera is constrained by the baseline and overcomes the limited depth range problem associated with SLAM for RGBD cameras. We first present the analytical feasibility for estimating the absolute scale through the fusion of 1D distance information and image information. Next, the analytical derivation of the laser-vision fusion is described in detail based on the local dense reconstruction of the image sequences. We also correct the scale drift of the monocular SLAM using the laser distance information which is independent of the drift error. Finally, application of this approach to both indoor and outdoor scenes is verified by the Technical University of Munich dataset of RGBD and self-collected data. We compare the effects of the scale estimation and drift correction of the proposed method with the SLAM for a monocular camera and a RGBD camera.

Light-field (LF) camera holds great promise for passive/general depth estimation benefited from high angular resolution, yet suffering small baseline for distanced region. While stereo solution with large baseline is superior to handle distant scenarios, the problem of limited angular resolution becomes bothering for near objects. Aiming for all-in-depth solution, we propose a cross-baseline LF camera using a commercial LF camera and a monocular camera, which naturally form a 'stereo camera' enabling compensated baseline for LF camera. The idea is simple yet non-trivial, due to the significant angular resolution gap and baseline gap between LF and stereo cameras.

An observer is developed that uses a monocular camera to reconstruct the positions of stationary features in a scene and the path the camera travels through the scene in real time. A Lyapunov based stability analysis is provided to show the observer is globally exponentially stable for less restrictive conditions than current literature. A simulation is provided to validate the developed theory and demonstrate performance.

Three-dimensional maps are useful for many applications; from the gaming industry, to augmented reality, to the development of tour guides of important landmarks such as museums or university campuses. The generation of such maps is very labour intensive and has therefore justified its automation using robots with range sensors such as lasers or cameras. This paper presents an automated 3D reconstruction system for indoor environments, which relies on a vision-based occupancy-grid SLAM (Simultaneous Localization and Mapping) to detect the ground. The novelty in our work is the method in which 3D information is extracted and fed to SLAM. Initially, coherent sections in the scene are segmented using a graph-cut algorithm, next 3D points extracted via a stereo camera are used to fit planes to each section. The ground plane is then determined based on the orientation of its normal and virtual rays are cast into the field of view from the camera center to the intersection of each ray's 2D projection with the Ground boundaries. Dense depth information can then be suggested from these rays and inputted to SLAM. Walls and ceiling are also built in a heuristic manner by satisfying normality constraints and keeping within the boundaries of Ground. Our system produces high-quality maps and reduces the high computational cost of dense stereo matching by processing only a sparse set of highly reliable salient features. Experiments are conducted inside a lab setting and results prove the success of the system.

At present, Chinese 3D reconstruction solutions using stereo cameras mainly face known, indoor, structured scenes; for the reconstruction of unstructured, larger-scale scenes with a large variety of texture information of different intensities, there are certain difficulties in ensuring accuracy and real-time processing. For the above problems, we propose a scene reconstruction method using stereo vision. Firstly, considering the influence of outdoor lighting and weather on the captured 2D images, the optimized SAD-FAST feature detection algorithm and stereo-matching strategy were employed in the stereo-matching stage to improve the overall efficiency and matching quality at this stage. Then, a homogenized feature extraction algorithm with gradient value decreasing step by step (GVDS) was used in the depth value calculation to ensure a sufficient number of feature points for strong texture information while extracting features from weak-texture areas, which greatly improved the quality and speed of unstructured scene reconstruction. We conducted experiments to validate the proposed method, and the results showed the feasibility of the proposed method and its high practical value.

Simultaneous Localization and Mapping (SLAM) technology can make the robot in the unknown area positioning and building the map. Aiming at the problem of the indoor positioning in a small area, SLAM algorithm based on monocular camera was used. Feature based method was introduced to estimate the position of the robot and build the local map, and then BA (Bundle Adjustment) and graph optimization was used to optimize in the back-end. In order to solve the problem of scale drift in positioning, the bag-of-words (BoW) model and loop closing were used to reduce the system error, and an overall map was constructed at last. The experiment results show that the proposed algorithm can construct a sparse map in a small indoor scene, and the positioning error in the x direction reaches 0.0528 m, which meets the accuracy requirements and effectively reduces the scale drift.

Abstract3D reconstruction of deformable (ornon‐rigid) scenes from a set of monocular 2D image observations is a long‐standing and actively researched area of computer vision and graphics. It is an ill‐posed inverse problem, since—without additional prior assumptions—it permits infinitely many solutions leading to accurate projection to the input 2D images. Non‐rigid reconstruction is a foundational building block for downstream applications like robotics, AR/VR, or visual content creation. The key advantage of using monocular cameras is their omnipresence and availability to the end users as well as their ease of use compared to more sophisticated camera set‐ups such as stereo or multi‐view systems. This survey focuses on state‐of‐the‐art methods for dense non‐rigid 3D reconstruction of various deformable objects and composite scenes from monocular videos or sets of monocular views. It reviews the fundamentals of 3D reconstruction and deformation modeling from 2D image observations. We then start from general methods—that handle arbitrary scenes and make only a few prior assumptions—and proceed towards techniques making stronger assumptions about the observed objects and types of deformations (e.g. human faces, bodies, hands, and animals). A significant part of this STAR is also devoted to classification and a high‐level comparison of the methods, as well as an overview of the datasets for training and evaluation of the discussed techniques. We conclude by discussing open challenges in the field and the social aspects associated with the usage of the reviewed methods.

Accurate vehicle ego-localization is key for autonomous vehicles to complete high-level navigation tasks. The state-of-the-art localization methods adopt visual and light detection and ranging (LiDAR) simultaneous localization and mapping (SLAM) to estimate the position of the vehicle. However, both of them may suffer from error accumulation due to long-term running without loop optimization or prior constraints. Actually, the vehicle cannot always return to the revisited location, which will cause errors to accumulate in Global Navigation Satellite System (GNSS)-challenged environments. To solve this problem, we proposed a novel localization method with prior dense visual point cloud map constraints generated by a stereo camera. Firstly, the semi-global-block-matching (SGBM) algorithm is adopted to estimate the visual point cloud of each frame and stereo visual odometry is used to provide the initial position for the current visual point cloud. Secondly, multiple filtering and adaptive prior map segmentation are performed on the prior dense visual point cloud map for fast matching and localization. Then, the current visual point cloud is matched with the candidate sub-map by normal distribution transformation (NDT). Finally, the matching result is used to update pose prediction based on the last frame for accurate localization. Comprehensive experiments were undertaken to validate the proposed method, showing that the root mean square errors (RMSEs) of translation and rotation are less than 5.59 m and 0.08°, respectively.

This paper describes a new sensor system for 3D environment perception using stereo structured infrared light sources and a camera. Environment and obstacle sensing is the key issue for mobile robot localization and navigation. Laser scanners and infrared scanners cover <TEX>$180^{\circ}$</TEX> and are accurate but too expensive. Those sensors use rotating light beams so that the range measurements are constrained on a plane. 3D measurements are much more useful in many ways for obstacle detection, map building and localization. Stereo vision is very common way of getting the depth information of 3D environment. However, it requires that the correspondence should be clearly identified and it also heavily depends on the light condition of the environment. Instead of using stereo camera, monocular camera and two projected infrared light sources are used in order to reduce the effects of the ambient light while getting 3D depth map. Modeling of the projected light pattern enabled precise estimation of the range. Two successive captures of the image with left and right infrared light projection provide several benefits, which include wider area of depth measurement, higher spatial resolution and the visibility perception.

We present a challenging new benchmark and learning-environment for robot learning: RLBench. The benchmark features 100 completely unique, hand-designed tasks, ranging in difficulty from simple target reaching and door opening to longer multi-stage tasks, such as opening an oven and placing a tray in it. We provide an array of both proprioceptive observations and visual observations, which include rgb, depth, and segmentation masks from an over-the-shoulder stereo camera and an eye-in-hand monocular camera. Uniquely, each task comes with an infinite supply of demos through the use of motion planners operating on a series of waypoints given during task creation time; enabling an exciting flurry of demonstration-based learning possibilities. RLBench has been designed with scalability in mind; new tasks, along with their motion-planned demos, can be easily created and then verified by a series of tools, allowing users to submit their own tasks to the RLBench task repository. This large-scale benchmark aims to accelerate progress in a number of vision-guided manipulation research areas, including: reinforcement learning, imitation learning, multi-task learning, geometric computer vision, and in particular, few-shot learning. With the benchmark's breadth of tasks and demonstrations, we propose the first large-scale few-shot challenge in robotics. We hope that the scale and diversity of RLBench offers unparalleled research opportunities in the robot learning community and beyond. Benchmarking code and videos can be found at https://sites.google.com/view/rlbench .

This paper describes an automatic background substitution for PC-softphone applications. Different from previous works on this topic which are mostly using stereo camera, we provide a solution which can segment the foreground region with monocular camera which is not moving. There are three main contributions from this paper. First, our proposed algorithm can segment the foreground region without any manual initialization. Second, we improve accuracy of the foreground segmentation using the Temporal Foreground Probability Model(TFPM). Finally, natural composite image was achieved by adaptive color and intensity correction method

A new era of mobile devices (phones and tablets) has emerged with some of them possessing a front camera facing the person or user and stereo cameras in the rear side. This paper emphasise on a novel method to use such devices with monocular camera to determine the depth between the user and the front camera using a back propagation neural network (BPNN). This depth is successively used to calculate the zooming factor for a legible view, to read a document on the display of the mobile device. In the absence of camera parameters and also considering the fact that the camera and user's face are in constant motion; measuring the distance of the face from the camera becomes a hard problem. We propose the use of frontal facial features acquired from the monocular camera to find the depth information with the use of supervised learning algorithm. Training the BPNN for the facial features for a standard face is one time factory programmed during camera manufacturing. One time new user's face registration for the gadget using rear cameras followed by the simulation of depth from the front camera using the trained BPNN is proposed in this work. The results of distance estimation are likened with stereo camera setup and the approach is validated.

In numerous applications, such as collision detection, Robotic handling, Robotic-based manufacturing facility, and Advanced Driver Assistance Systems (ADAS), depth estimation is crucial and the most significant task. Radar, Ultrasonic, Lidar technologies both operate by reflecting radio or sound wave or laser beams respectively. Stereo cameras used for depth estimation are costly and increase the cost of the system. There are a few Monocular camera depth estimate approaches that have evolved using mathematical calculation. One approach uses pixel to depth estimate mapping and the other uses the geometry of the road, the contact point of the vehicle on road, and camera properties. The proposed solution would implement depth detection using a monocular camera with deep learning. The objective of the proposed solution is to detect the depth from a monocular image and calibrate it with the actual depth. The pixel to distance data which derived from camera properties is used to run through varied hidden layers and nodes to conclude the implementation using Deep Learning (DL). Evidently, train and test data learn and converge more quickly on the deeper models which has more hidden layers and nodes.

A modular software framework design that allows flexible implementation of eye-in-hand sensing and motion control for agricultural robotics in dense vegetation is reported. Harvesting robots in cultivars with dense vegetation require multiple viewpoints and on-line trajectory adjustments in order to reduce the amount of false negatives and correct for fruit movement. In contrast to specialised software, the framework proposed aims to support a wide variety of agricultural use cases, hardware and extensions. A set of Robotic Operating System (ROS) nodes was created to ensure modularity and separation of concerns, implementing functionalities for application control, robot motion control, image acquisition, fruit detection, visual servo control and simultaneous localisation and mapping (SLAM) for monocular relative depth estimation and scene reconstruction. Coordination functionality was implemented by the application control node with a finite state machine. In order to provide visual servo control and simultaneous localisation and mapping functionalities, off-the-shelf libraries Visual Servoing Platform library (ViSP) and Large Scale Direct SLAM (LSD-SLAM) were wrapped in ROS nodes. The capabilities of the framework are demonstrated by an example implementation for use with a sweet-pepper crop, combined with hardware consisting of a Baxter robot and a colour camera placed on its end-effector. Qualitative tests were performed under laboratory conditions using an artificial dense vegetation sweet-pepper crop. Results indicated the framework can be implemented for sensing and robot motion control in sweet-pepper using visual information from the end-effector. Future research to apply the framework to other use-cases and validate the performance of its components in servo applications under real greenhouse conditions is suggested.

This paper presents a sensor fusion strategy applied for Simultaneous Localization and Mapping (SLAM) in dynamic environments. The designed approach consists of two features: (i) the first one is a fusion module which synthesizes line segments obtained from laser rangefinder and line features extracted from monocular camera. This policy eliminates any pseudo segments that appear from any momentary pause of dynamic objects in laser data. (ii) The second characteristic is a modified multi-sensor point estimation fusion SLAM (MPEF-SLAM) that incorporates two individual Extended Kalman Filter (EKF) based SLAM algorithms: monocular and laser SLAM. The error of the localization in fused SLAM is reduced compared with those of individual SLAM. Additionally, a new data association technique based on the homography transformation matrix is developed for monocular SLAM. This data association method relaxes the pleonastic computation. The experimental results validate the performance of the proposed sensor fusion and data association method.