3D reconstruction in orbital proximity operations

The PRISMA project constitutes an in-orbit test bed for Guidance, Navigation and Control (GNC) algorithms and sensors for advanced formation flying and rendezvous operations. Several experiments regarding manoeuvring and sensor experiments will be performed, sorted in 4 main categories – Autonomous Formation Flying, Homing and Rendezvous, Proximity Operations and Final Approach and Recede. The project consists of two satellites, one highly manoeuvrable Main satellite and one Target satellite without orbit control. The spacecraft masses are in the order of 150 and 50 kg respectively. The project is carried out by Swedish Space Corporation (SSC) in close cooperation with the German Aerospace Centre (DLR), CNES and the Danish Technical University. The project will also demonstrate 2 new propulsion technologies, the High Performance Green Propellant motor system and a MEMS technology microthruster cold-gas system milli-Newton thrust range. The launch is planned for the spring of 2009. The mission will be controlled from SSC facilities in Stockholm and Kiruna.

Pose estimation is critical for mobile robots to fulfill various tasks, such as path following or mapping the environment. This is usually accomplished by simultaneous localization and mapping (SLAM). However, computationally constrained systems, such as planetary rovers, rely on less intensive guidance navigation and control (GNC) solutions generally derived solely from visual odometry (VO), wheel odometry, and the onboard inertial measurement unit. Although providing adequate localization performances, the drift accumulated overtime is not compensated by loop closing capabilities, typical of SLAM. Usually, rovers send surface images to the ground station, and these images are used for multiple purposes, such as scientific and operational planning. The number of images is constrained by the communication bandwidth and power budget. The set of transmitted images can be used as a means to correct the robot’s trajectory in an off-line manner. In this work, a solution is presented to the problem of selecting the optimal set of viewpoints belonging to the planned path from which to capture and transmit images: 1) it guarantees accurate trajectory correction and 2) complies with the maximum number of images that can be transmitted to ground control given the available data budget. To this end, it is proposed: 1) a delocalized/decentralized sensor fusion approach based on pose graph optimization and structure from motion and 2) a strategy to select a minimal set of viewpoints along the trajectory that, given a tentative geometry of the environment and the global path that the rover must follow, minimizes the uncertainty of all the robot poses. Optimal camera viewpoint positions are selected as a function of the planned trajectory, the approximate scene geometry, and the maximum transmittable number of images. The proposed method has been tested on a dataset of stereo-images collected in a representative Martian environment, the ALTEC Mars Terrain Simulator (MTS), with the ExoMars testing rover (ExoTeR—European Space Agency, Paris, France, property). Rover stereo-images ground truth was given with millimetric accuracy by a motion capture (MC) system.

Three-dimensional maps are useful for many applications; from the gaming industry, to augmented reality, to the development of tour guides of important landmarks such as museums or university campuses. The generation of such maps is very labour intensive and has therefore justified its automation using robots with range sensors such as lasers or cameras. This paper presents an automated 3D reconstruction system for indoor environments, which relies on a vision-based occupancy-grid SLAM (Simultaneous Localization and Mapping) to detect the ground. The novelty in our work is the method in which 3D information is extracted and fed to SLAM. Initially, coherent sections in the scene are segmented using a graph-cut algorithm, next 3D points extracted via a stereo camera are used to fit planes to each section. The ground plane is then determined based on the orientation of its normal and virtual rays are cast into the field of view from the camera center to the intersection of each ray's 2D projection with the Ground boundaries. Dense depth information can then be suggested from these rays and inputted to SLAM. Walls and ceiling are also built in a heuristic manner by satisfying normality constraints and keeping within the boundaries of Ground. Our system produces high-quality maps and reduces the high computational cost of dense stereo matching by processing only a sparse set of highly reliable salient features. Experiments are conducted inside a lab setting and results prove the success of the system.

An integrated system composed of guidance, navigation and control (GNC) system for autonomous proximity operations and the docking of two spacecraft was developed. The position maneuvers were determined through the integration of the statedependent Riccati equation formulated from nonlinear relative motion dynamics and relative navigation using rendezvous laser vision (Lidar) and a vision sensor system. In the vision sensor system, a switch between sensors was made along the approach phase in order to provide continuously effective navigation. As an extension of the rendezvous laser vision system, an automated terminal guidance scheme based on the Clohessy-Wiltshire state transition matrix was used to formulate a “V-bar hopping approach” reference trajectory. A proximity operations strategy was then adapted from the approach strategy used with the automated transfer vehicle. The attitude maneuvers, determined from a linear quadratic Gaussian-type control including quaternion based attitude estimation using star trackers or a vision sensor system, provided precise attitude control and robustness under uncertainties in the moments of inertia and external disturbances. These functions were then integrated into an autonomous GNC system that can perform proximity operations and meet all conditions for successful docking. A sixdegree of freedom simulation was used to demonstrate the effectiveness of the integrated system.

Effective measurement of the reflective or transparent surface of an object has always been a disadvantage in laser scanning modeling. We propose a fast and complete three-dimensional (3D) reconstruction method for small static objects using laser scanning and the structure from motion (SFM) algorithm. Meanwhile, a complete reconstruction workflow is designed and a multi-angle 3d reconstruction system is set up. To generate the complete point cloud model of the object, the SFM algorithm is used to reconstruct the surface part of the object, the data for which cannot be obtained by the laser measuring instrument. The experimental results show that this method not only improves the speed, accuracy, integrity, and visual effect of 3D reconstruction of small objects, but also extends the scope of 3D reconstruction of laser measurement.

3D reconstruction can be used as a platform to monitor the performance of\nactivities on construction site, such as construction progress monitoring,\nstructure inspection and post-disaster rescue. Comparing to other sensors, RGB\nimage has the advantages of low-cost, texture rich and easy to implement that\nhas been used as the primary method for 3D reconstruction in construction\nindustry. However, the image-based 3D reconstruction always requires extended\ntime to acquire and/or to process the image data, which limits its application\non time critical projects. Recent progress in Visual Simultaneous Localization\nand Mapping (SLAM) make it possible to reconstruct a 3D map of construction\nsite in real-time. Integrated with Unmanned Aerial Vehicle (UAV), the obstacles\nareas that are inaccessible for the ground equipment can also be sensed.\nDespite these advantages of visual SLAM and UAV, until now, such technique has\nnot been fully investigated on construction site. Therefore, the objective of\nthis research is to present a pilot study of using visual SLAM and UAV for\nreal-time construction site reconstruction. The system architecture and the\nexperimental setup are introduced, and the preliminary results and the\npotential applications using Visual SLAM and UAV on construction site are\ndiscussed.\n

In modern medicine, 3D reconstruction plays an important role in improving the accuracy of disease detection and treatment especially in the process of robot-assisted surgery. Accordingly, the choice of 3D reconstruction method is of great importance, which provides real-time visualization. This study analyses and compares two types of methods in 3D medical image reconstruction and visualization: Surface Rendering and Volume Rendering. To be more specific, four algorithms including Contour Filter (CF), Marching Cubes (MC), Composite Volume Rendering (CVR) and Texture Mapping Hardware (TMH) were implemented and compared from the perspective of the rendering effect, speed and interactivity, in which the former two algorithms belong to the concept of Surface Rendering and the latter two algorithms are in the range of Volume Rendering. These algorithms were implemented by Microsoft Visual Studio with Visualization Toolkit (VTK) to reconstruct the 3D images of patients CT image data. The mechanism and realization of these algorithms were described, and the merits and shortcomings of these different rendering methods were compared. Based on the comparison results, the 3D reconstruction and rendering method can be determined according to the specific requirements of robot-assisted surgery.

3D human face model reconstruction is essential to the generation of facial animations that is widely used in the field of virtual reality (VR). The main issues of 3D facial model reconstruction based on images by vision technologies are in twofold: one is to select and match the corresponding features of face from two images with minimal interaction and the other is to generate the realistic-looking human face model. In this paper, a new algorithm for realistic-looking face reconstruction is presented based on stereo vision. Firstly, a pattern is printed and attached to a planar surface for camera calibration, and corners generation and corners matching between two images are performed by integrating modified image pyramid Lucas-Kanade (PLK) algorithm and local adjustment algorithm, and then 3D coordinates of corners are obtained by 3D reconstruction. Individual face model is generated by the deformation of general 3D model and interpolation of the features. Finally, realistic-looking human face model is obtained after texture mapping and eyes modeling. In addition, some application examples in the field of VR are given. Experimental result shows that the proposed algorithm is robust and the 3D model is photo-realistic.

Structure and diversity of human dendritic spines evidenced by a new three-dimensional reconstruction procedure for Golgi staining and light microscopy

Focused ion beam scanning electron microscopy is well suited for volumetric extractions and 3D reconstructions of plant cells and its organelles. The three-dimensional (3D) reconstruction of individual plant cells is an important tool to extract volumetric data of organelles and is necessary to fully understand ultrastructural changes and adaptations of plants to their environment. Methods such as the 3D reconstruction of cells based on light microscopical images often lack the resolution necessary to clearly reconstruct all cell compartments within a cell. The 3D reconstruction of cells through serial sectioning transmission electron microscopy (ssTEM) and focused ion beam scanning electron microscopy (FIB-SEM) are powerful alternatives but not widely used in plant sciences. Here, we present a method for the 3D reconstruction and volumetric extraction of plant cells based on FIB milling and compare the results with 3D reconstructions obtained with ssTEM. When compared to 3D reconstruction based on ssTEM, FIB-SEM delivered similar results. The data extracted in this study demonstrated that tobacco cells were larger (31410 µm3) than pumpkin cells (20697 µm3) and contained more chloroplasts (175 vs. 124), mitochondria (1317 vs. 291) and peroxisomes (745 vs. 79). While individual chloroplasts, mitochondria, peroxisomes were larger in pumpkin plants (25, 53, and 50%, respectively) they covered more total volume in tobacco plants (5390, 395, 374 µm3, respectively) due to their higher number per cell when compared to pumpkin plants (4762, 134, 59 µm3, respectively). While image acquisition with FIB-SEM was automated, software controlled, and less difficult than ssTEM, FIB milling was slower and sections couldnot be revised or re-imaged as they were destroyed by the ion beam. Nevertheless, the results in this study demonstrated that both, FIB-SEM and ssTEM, are powerful tools for the 3D reconstruction of and volumetric extraction from plant cells and that there were large differences in size, number, and organelle composition between pumpkin and tobacco cells.

In-water visual ship hull inspection using unmanned underwater vehicles needs to be performed at very close range to the target surface because of the visibility limitations in underwater environments mainly due to light attenuation, scattering, and water turbidity. These environmental challenges result in ineffective photometric and geometric information in hull surface images and, therefore, the performance of conventional three-dimensional (3D) reconstruction techniques is often unsatisfactory. This paper addresses a visual mapping method for 3D reconstruction of underwater ship hull surface using a monocular camera as a primary mapping sensor. The main idea of the proposed approach is to model the moderately curved hull surface as a combination of piecewise-planar panels, and to generate a global map by aligning the local images in a two-dimensional reference frame and correcting them appropriately to reflect the information of perspective projections of the 3D panels. The estimated 3D panels associated with the local images are used to extract the loop-closure relative measurements in the framework of simultaneous localization and mapping (SLAM) for precise camera trajectory estimation and 3D reconstruction results. The validity and practical feasibility of the proposed method are demonstrated using a dataset obtained in a field experiment with a full-scale ship in a real sea environment.

In the last decades, Simultaneous Localization and Mapping (SLAM) proved to be a fundamental topic in the field of robotics, due to the many applications, ranging from autonomous driving to 3D reconstruction. Many systems have been proposed in literature exploiting a heterogeneous variety of sensors. State-of-the-art methods build their own map from scratch, using only data coming from the equipment of the robot, and not exploiting possible reconstructions of the environment. Moreover, temporary loss of data proves to be a challenge for SLAM systems, as it demands efficient re-localization to continue the localization process. In this paper, we present a SLAM system that exploits additional information coming from mapping services like OpenStreetMaps, hence the name OSM-SLAM, to face these issues. We extend an existing LiDAR-based Graph SLAM system, ART-SLAM, making it able to integrate the 2D geometry of buildings in the trajectory estimation process, by matching a prior OpenStreetMaps map with a single LiDAR scan. Each estimated pose of the robot is then associated with all buildings surrounding it. This association allows to improve localization accuracy, but also to adjust possible mistakes in the prior map. The pose estimates coming from SLAM are then jointly optimized with the constraints associated with the various OSM buildings, which can assume one of the following types: Buildings are always fixed (Prior SLAM); buildings surrounding a robot are movable in chunks, for every scan (Rigid SLAM); and every single building is free to move independently from the others (Non-rigid SLAM). Lastly, OSM maps can also be used to re-localize the robot when sensor data is lost. We compare the accuracy of the proposed system with existing methods for LiDAR-based SLAM, including the baseline, also providing a visual inspection of the results. The comparison is made by evaluating the estimated trajectory displacement using the KITTI odometry dataset. Moreover, the experimental campaign, along with an ablation study on the re-localization capabilities of the proposed system and its accuracy in loop detection-denied scenarios, allow a discussion about how the quality of prior maps influences the SLAM procedure, which may lead to worse estimates than the baseline.

With the advancement of Light Detection and Ranging (LiDAR) technology and computer science, LiDAR–Inertial Simultaneous Localization and Mapping (SLAM) has become essential in autonomous driving, robotic navigation, and 3D reconstruction. However, dynamic objects such as pedestrians and vehicles, with complex terrain conditions, pose serious challenges to existing SLAM systems. These factors introduce artifacts into the acquired point clouds and result in significant vertical drift in SLAM trajectories. To address these challenges, this study focuses on controlling vertical drift errors in LiDAR–Inertial SLAM systems operating in dynamic environments. The research focuses on three key aspects: ground point segmentation, dynamic artifact removal, and vertical drift optimization. In order to improve the robustness of ground point segmentation operations, this study proposes a method based on a concentric sector model. This method divides point clouds into concentric regions and fits flat surfaces within each region to accurately extract ground points. To mitigate the impact of dynamic objects on map quality, this study proposes a removal algorithm that combines multi-frame residual analysis with curvature-based filtering. Specifically, the algorithm tracks residual changes in non-ground points across consecutive frames to detect inconsistencies caused by motion, while curvature features are used to further distinguish moving objects from static structures. This combined approach enables effective identification and removal of dynamic artifacts, resulting in a reduction in vertical drift.

Nowadays, the use of RGB-D sensors have focused a lot of research in computer vision and robotics. These kinds of sensors, like Kinect, allow to obtain 3D data together with color information. However, their working range is limited to less than 10 meters, making them useless in some robotics applications, like outdoor mapping. In these environments, 3D lasers, working in ranges of 20-80 meters, are better. But 3D lasers do not usually provide color information. A simple 2D camera can be used to provide color information to the point cloud, but a calibration process between camera and laser must be done. In this paper we present a portable calibration system to calibrate any traditional camera with a 3D laser in order to assign color information to the 3D points obtained. Thus, we can use laser precision and simultaneously make use of color information. Unlike other techniques that make use of a three-dimensional body of known dimensions in the calibration process, this system is highly portable because it makes use of small catadioptrics that can be placed in a simple manner in the environment. We use our calibration system in a 3D mapping system, including Simultaneous Location and Mapping (SLAM), in order to get a 3D colored map which can be used in different tasks. We show that an additional problem arises: 2D cameras information is different when lighting conditions change. So when we merge 3D point clouds from two different views, several points in a given neighborhood could have different color information. A new method for color fusion is presented, obtaining correct colored maps. The system will be tested by applying it to 3D reconstruction.

In this work, we present a comparative analysis of the trajectories estimated from various Simultaneous Localization and Mapping (SLAM) systems in a simulation environment for vineyards. Vineyard environment is challenging for SLAM methods, due to visual appearance changes over time, uneven terrain, and repeated visual patterns. For this reason, we created a simulation environment specifically for vineyards to help studying SLAM systems in such a challenging environment. We evaluated the following SLAM systems: LIO-SAM, StaticMapping, ORB-SLAM2, and RTAB-MAP in four different scenarios. The mobile robot used in this study equipped with 2D and 3D lidars, IMU, and RGB-D camera (Kinect v2). The results show good and encouraging performance of RTAB-MAP in such an environment.