Unsupervised Monocular Depth Reconstruction of Non-Rigid Scenes

Generating dynamic scenes from images has gained increasing attention. Existing methods have two major limitations: 1) they can hardly handle sparse images which exhibit limited geometry constraints and insufficient motion; 2) they struggle to maintain spatial-temporal consistency when rendering multi-view videos. To address these limitations, we propose SCSV, a spatial-temporal consistent dynamic scene generation method from sparse views. Our method consists of two stages: scene reconstruction and scene expansion, both of which decouple background and foreground. In the scene reconstruction stage, we first interpolate a set of images between the input images based on a video generation model, followed by the optimization of the scene Gaussian from the interpolated and input images. To improve the spatial-temporal consistency of the reconstructed scene, we propose an uncertainty-aware Gaussian training approach, which introduces adaptive weights of images and pixels. In the scene expansion stage, for background, we render novel views and refine them with a geometry-aware diffusion process. These refined images are then used to incrementally add the Gaussians. As to foreground, we generate human motion according to previous motion, enabling temporal coherent generation of motion. To further enhance the physical plausibility, we integrate the expanded foreground into the background using a gravity-aware alignment. Experiments on NeuMan, Bonn, and EMDB datasets demonstrate that our SCSV achieves superior performance compared to state-of-the-art methods. The code will be released upon acceptance.

This paper addresses the challenge that current event-based video reconstruction methods cannot produce static background information. Recent research has uncovered the potential of event cameras in capturing static scenes. Nonetheless, image quality deteriorates due to noise interference and detail loss, failing to provide reliable background information. We propose a two-stage reconstruction strategy to address these challenges and reconstruct static scene images comparable to frame cameras. Building on this, we introduce the URSEE framework designed for reconstructing motion videos with static backgrounds. This framework includes a parallel channel that can simultaneously process static and dynamic events, and a network module designed to reconstruct videos encompassing both static and dynamic scenes in an end-to-end manner. We also collect a real-captured dataset for static reconstruction, containing both indoor and outdoor scenes. Comparison results indicate that the proposed method achieves state-of-the-art performance on both synthetic and real data.

With the development of enterprise intelligence and automation, the inspection of power distribution room is not limited to manual work. This paper proposes a semantic segmentation method based on improved ORB-SLAM3 and YOLOV5 fusion to eliminate dynamic features. ORB-SLAM3 is used to realize scene reconstruction to generate dense point cloud maps. In this paper, all the differences between ORB-SLAM2 and ORBSLAM3 are compared, and the dense point cloud generation and loop detection modules are added to make the mapping more accurate. In real scenes, the depth sensor is inevitably disturbed by dynamic objects when obtaining real environmental information. Compared with dynamic SLAM for object detection, the semantic segmentation module of YOLOV5 is used in this paper, which can remove dynamic objects while minimizing the loss of effective features. In addition, TensorRT is used to accelerate the construction of the map, and the semantic map is generated, which makes a prerequisite for the subsequent conversion of the map into a two-dimensional occupancy map. Experimental results show that compared with the original ORB-SLAM2/3, the improved ORBSLAM3 greatly improves the accuracy in dynamic scenes. A more real-time and accurate scene reconstruction function of power distribution room is realized.

Accurate reconstruction of dynamic objects in urban scenes remains a challenging task. Existing methods typically rely on complex 3D annotations to identify dynamic objects, which are expensive and difficult to obtain. In this paper, we propose a novel approach to address this challenge by leveraging accessible and effective 2D annotations. We utilize a 2D foundation model to identify dynamic objects within the image and lift this knowledge to 3D space, obtaining the 3D point cloud of dynamic objects. To capture the rigid motion patterns of dynamic objects, we introduce an object-level motion model together with the local aggregation strategy. Additionally, we propose a structural consistency constraint to maintain the consistency of dynamic objects during motion, significantly improving the accuracy of scene reconstruction. Extensive experiments demonstrate that our method achieves comparable reconstruction performance to methods relying on 3D annotations, providing a cost-effective and accurate solution for dynamic urban scene reconstruction.

Commodity depth cameras, such as the Microsoft Kinect®, have been widely used for the capture and reconstruction of the 3D structure of room-sized dynamic scenes. Camera placement and coverage during capture significantly impact the quality of the resulting reconstruction. In particular, dynamic occlusions and sensor interference have been shown to result in poor resolution and holes in the reconstruction results. This paper presents a novel algorithmic framework and a method for off-line optimization of depth cameras placements for a given 3D dynamic scene, simulated using virtual 3D models. We derive a fitness metric for a particular configuration of sensors by combining factors such as visibility and resolution of the entire dynamic scene with probabilities of interference between sensors. We employ this fitness metric both in a greedy algorithm that determines the number of depth cameras needed to cover the scene, and in a simulated annealing algorithm that optimizes the placements of those sensors. We compare our algorithm's optimized placements with manual sensor placements for a real dynamic scene. We present quantitative assessments using our fitness metric, as well as qualitative assessments to demonstrate that our algorithm not only enhances the resolution and total coverage of the reconstruction, but also fills in voids by avoiding occlusions and sensor interference when compared with the reconstruction of the same scene using mual sensor placement.

Criminator: An Easy-to-Use XR "Crime Animator" for Rapid Reconstruction and Analysis of Dynamic Crime Scenes

Chapter 2 - Multi-View Calibration, Synchronization, and Dynamic Scene Reconstruction

Single-shot structured light methods allow 3D reconstruction of dynamic scenes. However, such methods lose spatial resolution and perform poorly around depth discontinuities. Previous single-shot methods project the same pattern repeatedly; thereby spatial resolution is reduced even if the scene is static or has slowly moving parts. We present a structured light system using a sequence of shifted stripe patterns that is decodable both spatially and temporally. By default, our method allows single-shot 3D reconstruction with any of our projected patterns by using spatial windows. Moreover, the sequence is designed so as to progressively improve the reconstruction quality around depth discontinuities by using temporal windows.Our method enables motion-aware reconstruction for each pixel: The best spatio-temporal window is automatically selected depending on the scene structure, motion, and the number of available images. This significantly reduces the number of pixels around discontinuities where depth cannot be recovered in traditional approaches. Our decoding scheme extends the adaptive window matching commonly used in stereo by incorporating temporal windows with 1D spatial windows. We demonstrate the advantages of our approach for a variety of scenarios including thin structures, dynamic scenes, and scenes containing both static and dynamic regions.KeywordsStructured lightmotion-aware 3D reconstructionspatio-temporal decodingadaptive window matching

Representing and rendering dynamic underwater scenes present significant challenges due to the medium’s inherent properties, which result in image blurring and information ambiguity. To overcome these challenges and accomplish real-time rendering of dynamic underwater environments while maintaining efficient training and storage, we propose Underwater Dynamic Scene Reconstruction Gaussian Splatting (UDR-GS), a method based on Gaussian Splatting. By leveraging prior information from a pre-trained depth estimation model and smoothness constraints between adjacent images, our approach uses the estimated depth as a geometric prior to aid in color-based optimization, significantly reducing artifacts and improving geometric accuracy. By integrating depth guidance into the Gaussian Splatting (GS) optimization process, we achieve more precise geometric estimations. To ensure higher stability, smoothness constraints are applied between adjacent images, maintaining consistent depth for neighboring 3D points in the absence of boundary conditions. The symmetry concept is inherently applied in our method by maintaining uniform depth and color information across multiple viewpoints, which enhances the reconstruction quality and visual coherence. Using 4D Gaussian Splatting (4DGS) as a baseline, our strategy demonstrates superior performance in both RGB novel view synthesis and 3D geometric reconstruction. On average, across multiple datasets, our method shows an improvement of approximately 1.41% in PSNR and a 0.75% increase in SSIM compared with the baseline 4DGS method, significantly enhancing the visual quality and geometric fidelity of dynamic underwater scenes.

We propose a content-based 3D mosaic representation for long video sequences of 3D and dynamic scenes captured by a camera on a mobile platform. The motion of the camera has a dominant direction of motion (as on an airplane or ground vehicle), but 6 degrees-of-freedom (DOF) motion is allowed. In the first step, a pair of generalized parallel-perspective (pushbroom) stereo mosaics is generated that captured both the 3D and dynamic aspects of the scene under the camera coverage. In the second step, a segmentation-based stereo matching algorithm is applied to extract parametric representation of the color, structure and motion of the dynamic and/or 3D objects in urban scenes where a lot of planar surfaces exist. Based on these results, the content-based 3D mosaic (CB3M) representation is created, which is a highly compressed visual representation for very long video sequences of dynamic 3D scenes. Experimental results are given

Efficient edge-aware surface mesh reconstruction for urban scenes

This paper describes a system for dense depth estimation from multiple color images in real-time. Our algorithm runs almost entirely on standard graphics hardware, leaving the main CPU free for other tasks such as image capture and higher level recognition. We follow a plane-sweep approach extended by truncated SSD scores, spatially shiftable windows and best camera selection to handle discontinuities. We do not need specialized hardware and exploit the computational power of freely programmable PC GPU hardware. Dense depth maps are computed with up to 20 fps.

The process of recovering dense depth maps from sparse depth information is prone to edge blurring. This paper proposes an image-guided depth completion algorithm to address this issue. The method uses the edges of the color image as prior constraints to construct an image topography map as an intermediate representation and performs nonlinear adaptive reconstruction based on the image content to adjust the position and scale of the pixel-weighted neighborhood. This approach avoids incorporating depth information with different distributions when estimating missing values, resulting in a full-resolution dense depth map with sharp edges. We conducted a quantitative comparison with state-of-the-art models on the KITTI and MidAir datasets, demonstrating that our algorithm has better performance and robustness in terms of completion accuracy. We also analyzed the impact of sparsity on the algorithm’s performance and its ability to recover fine structures in dense depth results and demonstrated the reconstruction results for sparse data in real-world scenarios.

The paper briefly overviews the design and applications of cylindrical panoramic cameras characterized by a rotating linear sensor capturing one image column at time. The camera provides very high image resolutions paid by motion distortions in dynamic scenes. The images are used for stereo reconstruction and visualization of static scenes when extremely high image resolution is of benefit.

In this paper, we present a method to reconstruct the world and multiple dynamic humans in 3D from a monocu-lar video input. As a key idea, we represent both the world and multiple humans via the recently emerging 3D Gaussian Splatting (3D-GS) representation, enabling to conveniently and efficiently compose and render them together. In par-ticular, we address the scenarios with severely limited and sparse observations in 3D human reconstruction, a common challenge encountered in the real world. To tackle this chal-lenge, we introduce a novel approach to optimize the 3D-GS representation in a canonical space by fusing the sparse cues in the common space, where we leverage a pre-trained 2D diffusion model to synthesize unseen views while keeping the consistency with the observed 2D appearances. We demon-strate our method can reconstruct high-quality animatable 3D humans in various challenging examples, in the presence of occlusion, image crops, few-shot, and extremely sparse observations. After reconstruction, our method is capable of not only rendering the scene in any novel views at arbitrary time instances, but also editing the 3D scene by removing individual humans or applying different motions for each human. Through various experiments, we demonstrate the quality and efficiency of our methods over alternative existing approaches.