Real-time underwater 3D scene reconstruction using commercial depth sensor

We present a novel framework named NeuralRecon for real-time 3D scene reconstruction from a monocular video. Unlike previous methods that estimate single-view depth maps separately on each key-frame and fuse them later, we propose to directly reconstruct local surfaces represented as sparse TSDF volumes for each video fragment sequentially by a neural network. A learning-based TSDF fusion module based on gated recurrent units is used to guide the network to fuse features from previous fragments. This design allows the network to capture local smoothness prior and global shape prior of 3D surfaces when sequentially reconstructing the surfaces, resulting in accurate, coherent, and real-time surface reconstruction. The fused features can also be used to predict semantic labels, allowing our method to reconstruct and segment the 3D scene simultaneously. Furthermore, we purpose an efficient self-supervised fine-tuning scheme that refines scene geometry based on input images through differentiable volume rendering. This fine-tuning scheme improves reconstruction quality on the fine-tuned scenes, as well as the generalization to similar test scenes. The experiments on ScanNet, 7-Scenes and Replica datasets show that our system outperforms state-of-the-art methods in terms of both accuracy and speed.

Real-time 3D reconstruction of static scenes can be achieved based on the RGB-D image sequence fusion. It is a popular practice to divide the space into uniform voxels and use a truncated signed distance function to represent surface information. In order to represent a scene of large scale, the voxel hash algorithm which stores voxels compressively can be used, but most of the conventional methods do not consider the complexity and roughness of the object surface in the scene, so the scene is represented with a uniform resolution. It somewhat limits the range of scene representation and the speed of real-time reconstruction. In this paper, a large-scale scene reconstruction algorithm based on voxel hashing storage with LOD representation is proposed. The main contributions include the following two aspects: (1) By preprocessing the depth image with smooth filtering, which ensures the accuracy of the data, it can effectively reduce the distortion caused by the sensor itself and violent motion and provide better support for the stages of voxel hashing, model rendering, and frame-to-model camera position tracking. (2) The 3D reconstruction with LOD representation is realized. We take the view distance and the roughness of the model surface as criteria to control the adaptive division and representation of spatial voxel blocks. Finally, we carried out qualitative and quantitative evaluations of the algorithm, and confirmed that the algorithm can achieve real-time reconstruction with different levels of detail in the commercial graphics processing hardware environment, and achieve a good fusion effect in large-scale scenes.

In this paper, we propose possibility for reconstruction of surface of an underwater object or 3D scene reconstruction of an underwater environment using an economical RGB-D sensor such as Microsoft Kinect. Reconstructing the 3D surface of an underwater object is a challenging task due to degraded quality of underwater images. There are various reasons of quality degradation of underwater images i.e., non-uniform illumination of light on the surface of objects, scattering and absorption effects. Particles and impurities present in underwater produces Gaussian noise on the captured underwater optical images which degrades the quality of images. However, using depth sensors, as a cost effective alternative, we aim to show that underwater 3D scene reconstruction is possible with sight tradeoffs on accuracy but major cost saving. The acquired depth data is proposed to be processed by applying real-time mesh generating techniques from the acquired point cloud. The experimental result aims to show that the proposed method reconstructs 3D surface of underwater objects accurately using captured underwater depth images.

Real-time 3D unstructured environment reconstruction utilizing VR and Kinect-based immersive teleoperation for agricultural field robots

Real-time indoor scene reconstruction aims to recover the 3D geometry of an indoor scene in real time with a sensor scanning the scene. Previous works of this topic consider pure static scenes, but in this paper, we focus on more challenging cases that the scene contains dynamic objects, for example, moving people and floating curtains, which are quite common in reality and thus are eagerly required to be handled. We develop an end-to-end system using a depth sensor to scan a scene on the fly. By proposing a Sigmoid-based Iterative Closest Point (S-ICP) method, we decouple the camera motion and the scene motion from the input sequence and segment the scene into static and dynamic parts accordingly. The static part is used to estimate the camera rigid motion, while for the dynamic part, graph node-based motion representation and model-to-depth fitting are applied to reconstruct the scene motions. With the camera and scene motions reconstructed, we further propose a novel mixed voxel allocation scheme to handle static and dynamic scene parts with different mechanisms, which helps to gradually fuse a large scene with both static and dynamic objects. Experiments show that our technique successfully fuses the geometry of both the static and dynamic objects in a scene in real time, which extends the usage of the current techniques for indoor scene reconstruction.

Tele-immersive systems enable 3D interactive experience in a virtual space by bringing together objects born in physical and virtual reality environments that are geographically distributed. The immersive and interactive experiences are achieved by fusing real-time color plus depth video of physical scenes from multiple stereo cameras, displaying 3D reconstructions of physical and virtual objects, and tracking intersections of objects in a virtual space to facilitate interactions between objects. While tele-immersive (TI) systems have been attracting a lot of attention these days as a possible advancement of the current 2D video-based communication systems and of the virtual reality spaces, the challenges lie in limited resources available to satisfy the many requirements on a usable TI system. One of the TI challenges remains the quality of real-time 3D reconstruction of physical scenes. Our work is motivated by the lack of understanding of the importance of various TI system parameters on the quality which is critical for allocating limited resources to make TI systems useful. This paper investigates the impact of different quality parameters of TI systems on accuracy and speed of task executions in TI environments. We have designed a task of moving a virtual ball from an initial center position to a basketball basket close to top in a closed virtual cuboid space. Successful completion of the task involves localization, orientation and motion coordination for a human in TI environment. We report quantitative and qualitative measurements collected during task executions under simulated variations of 3D reconstruction quality parameters. Based on our experimental results with 29 subjects under 13 quality variations, we concluded that the most important parameter is the frame rate followed by the presence of of a human rendering rather than just an avatar representation of hands.

Mapping of underwater environments is a critical task for a range of activities from monitoring coral reef habitats to surveying submerged archaeological sites. While recent advances in methods for terrestrial mapping can achieve dense 3D reconstructions of scenes in real-time, there remains the challenge of transferring these methods to the underwater domain due to characteristic effects on propagation of light through the water column that violate the brightness constancy constraint used in terrestrial techniques. Current state-of-the-art methods for underwater 3D reconstruction exploit a physical model of light propagation underwater to account for such range-dependent effects as scattering and attenuation; however, these methods necessitate careful calibration of attenuation coefficients required by the physical model, or rely on rough estimates of these coefficients from prior lab experiments. The main contribution of this paper is to develop a novel method to achieve simultaneous estimation of attenuation coefficients for color correction during structure recovery of an underwater scene by integrating this estimation directly into the bundle adjustment step, which performs non-linear optimization. To validate the proposed method, an artificial scene is submerged in a pure water tank and surveyed with a stereo camera platform to simulate an underwater robotic survey in a controlled environment. The target structure is imaged in air with an RGB-D sensor to provide ground truth structure and color, and a color calibration board is place in the scene for further reference. Results show that the proposed method can automatically estimate a water-column aware model for color correction of underwater images simultaneously to 3D reconstruction of the submerged scene.

3D or multi-dimensional medical image reconstruction is attracting more and more attention in modern era. Rapid volumetric image reconstruction is of paramount importance to clinicians for prompt diagnosis and analysis of complex tissue alternations. However, due to a high demand on computation for either the analytical or iterative image reconstruction algorithms, it is almost impossible to meet the requirement of real-time reconstruction. On the other hand, with high computing capability and broad banding characteristic, recently multi-core processors provide a new possibility and hopeful start for some of the most computation-demanding applications, such as medical imaging, digital media and entertainment equipment. This study aims to parallelize an analytical 3D CT image reconstruction algorithm, the so-called FDK method, and to implement it on an existing multi-core platform system, towards a real-time 3D medical image reconstruction. In this paper, the parallelization of the 3D image reconstruction algorithm, the coding optimizations and a comparative result of run time savings are described.

Methods of 3D reconstruction of medical images mainly included surface and volume reconstruction at present. In the experiment of 3D ultrasound reconstruction for spine of people, the result using volume reconstruction was unsatisfactory because of the ultrasound images' characteristic. So, we brought forward a method which was based on contour line to reconstruct 3D surface of spine. In this paper, we collected spinal ultrasound images using 3D freehand ultrasound firstly; next, tracing spinal contour line of each ultrasound images manually; finally, we used these contour lines to reconstruct spinal surface. The result showed that spinal surface reconstruction based on 2D contour line was close to real spinal surface, it could give physicians a better help in diagnosis of the spine.

Three-dimensional maps are useful for many applications; from the gaming industry, to augmented reality, to the development of tour guides of important landmarks such as museums or university campuses. The generation of such maps is very labour intensive and has therefore justified its automation using robots with range sensors such as lasers or cameras. This paper presents an automated 3D reconstruction system for indoor environments, which relies on a vision-based occupancy-grid SLAM (Simultaneous Localization and Mapping) to detect the ground. The novelty in our work is the method in which 3D information is extracted and fed to SLAM. Initially, coherent sections in the scene are segmented using a graph-cut algorithm, next 3D points extracted via a stereo camera are used to fit planes to each section. The ground plane is then determined based on the orientation of its normal and virtual rays are cast into the field of view from the camera center to the intersection of each ray's 2D projection with the Ground boundaries. Dense depth information can then be suggested from these rays and inputted to SLAM. Walls and ceiling are also built in a heuristic manner by satisfying normality constraints and keeping within the boundaries of Ground. Our system produces high-quality maps and reduces the high computational cost of dense stereo matching by processing only a sparse set of highly reliable salient features. Experiments are conducted inside a lab setting and results prove the success of the system.

The fusion of light detection and ranging (LiDAR) and camera provides comprehensive information about the depth and color texture of objects, finding wide applications in autonomous driving, three-dimensional (3D) scene reconstruction, and other fields. However, point clouds acquired via LiDAR are susceptible to noise, impacting downstream tasks like surface reconstruction and object recognition demanding high-quality point cloud outputs from 3D imaging systems. This paper proposes a novel 3D fusion imaging technique that enhances the quality of the fused point clouds from three distinct aspects, improving the accuracy and precision of point clouds by employing matched filter method, designing a preprocessing workflow for point clouds and images to enable their effective fusion, and smoothing noise and accentuating texture features using a filter algorithm on fused information. The method of matched filtering has been employed to reconstruct object distance information from photon count-time histograms, resulting in an improvement of approximately 120% compared to the method of directly calculating the average photon flight time. The point cloud filter proposed utilizing fused information demonstrates superior denoising capabilities compared to the state-of-the-art and has the capability to cluster points with similar features to highlight the texture characteristics of the point cloud.

With the increasing use of image-based 3D reconstruction in medical procedures, accurate scene reconstruction plays a crucial role in surgical navigation and assisted treatment. However, the monotonous colors, limited image features, and obvious brightness fluctuations of thoracoscopic scenes make the feature point matching process, on which traditional 3D reconstruction methods rely, unstable and unreliable. It brings a great challenge to accurate 3D reconstruction. In this study, a new method for implicit 3D reconstruction of monocular thoracoscopic scenes is proposed. The method combines a pre-trained metric depth estimation model with neural radiation field (NeRF) technique and uses dense SLAM to accurately compute the camera pose. To ensure the accuracy of the depth values and the structural consistency of the reconstructed scene, depth and normal constraints are added to the original color constraints of the NeRF network to achieve high-quality scene reconstruction results. We conducted experiments on the SCARED dataset and the clinical dataset. After comparing with other methods, the depth estimation accuracy and point cloud reconstruction quality of this paper outperform the existing methods. The method in this paper can provide more accurate 3D reconstruction of complex thoracic surgical scenes, which can significantly improve the accuracy and therapeutic efficacy of surgical navigation.

This article presents an overview over the whole process of generating a precise and rich 3D representation of the local environment of a moving mobile outdoor robot. The resulting model of this process is a camera image textured triangle mesh which is triangulated from a motion-corrected laser scanned 3D point cloud. The demanding requirements of autonomous off-road environment model acquisition are handled by applying a multi-sensor fusion approach. Additionally to the model creation process a novel way of detecting and describing feature points on a piece-wise linear 3D surfaces is presented. The set of feature points generated by the proposed method is a valuable abstraction of a whole outdoor scene that can be used in several following processing steps like mesh simplification or segmentation and robotics-specific tasks like obstacle detection or classification. All described methods are implemented and successfully used on the award-winning robot AMOR.

In this paper we introduce a new recommender system for urban tourists. The goal of the system is to enrich tourists’ experience by offering them personalised tour recommendations tailored to their dynamic user profiles. Particular attention in the proposed approach is paid to the influence of basic leisure needs of an individual, which include new experiences, entertainment, being in open area, relaxation, physical exercise, and socialising, on the tour composition. These needs tend to be dynamic and give rise to saturation effects and variety seeking behaviour. The system is developed as part of the larger c-Space framework, in which a number of technologies, such as projective augmented reality, a newly proposed near real-time 4D dynamic scene reconstruction, and affective computing, are brought together and used to enrich experiences of users in their interactions with built environments. The paper describes the main concepts of the recommender system and its implementation in the specified context of the city of Trento, Italy.

This paper attempts to print irregular 3D objects on a freeform surface. Notably, it is inevitable to cause errors when creating a specific object shape in simulated and real-life scenarios. This is mainly attributed to the unique characteristics of the materials adopted, which tend to develop different deformations properties. Nevertheless, since the simulated model provides a clue regarding the coordinates of the deformed shape, it is adopted as an essential indicator in estimating accurate real-world coordinates. Concretely, this paper presents a complete system that is capable of reconstructing the detailed surface coordinates in real-time conditions. To verify the effectiveness of the proposed method, a metal board is used as the primary material to create different curvatures physically and virtually. In particular, the real-time performance of the overall 3D surface reconstruction has been experimentally evaluated using several tools, such as KUKA KR90 R3100 robots, HoloLens 2, displacement sensor, and ArUco markers. Consequently, both the quantitative and qualitative results are presented to demonstrate the feasibility of the proposed method. Furthermore, the experimental data obtained manifest that there are significant differences between the simulated and the real-world coordinates. Thus, the findings of this study provide an insightful outlook and lay down several important implications for future practice.