Chapter 2 - Multi-View Calibration, Synchronization, and Dynamic Scene Reconstruction

This paper presents an approach to reconstruct non-stationary, articulated objects from silhouettes obtained with a monocular video sequence. We introduce the concept of motion blurred scene occupancies, a direct analogy of motion blurred images but in a 3D object scene occupancy space resulting from the motion/deformation of the object. Our approach starts with an image based fusion step that combines color and silhouette information from multiple views. To this end we propose to use a novel construct: the temporal occupancy point (TOP), which is the estimated 3D scene location of a silhouette pixel and contains information about duration of time it is occupied. Instead of explicitly computing the TOP in 3D space we directly obtain itpsilas imaged(projected) locations in each view. This enables us to handle monocular video and arbitrary camera motion in scenarios where complete camera calibration information may not be available. The result is a set of blurred scene occupancy images in the corresponding views, where the values at each pixel correspond to the fraction of total time duration that the pixel observed an occupied scene location. We then use a motion de-blurring approach to de-blur the occupancy images. The de-blurred occupancy images correspond to a silhouettes of the mean/motion compensated object shape and are used to obtain a visual hull reconstruction of the object. We show promising results on challenging monocular datasets of deforming objects where traditional visual hull intersection approaches fail to reconstruct the object correctly.

Monocular depth reconstruction of complex and dynamic scenes is a highly challenging problem. While for rigid scenes learning-based methods have been offering promising results even in unsupervised cases, there exists little to no literature addressing the same for dynamic and deformable scenes. In this work, we present an unsupervised monocular framework for dense depth estimation of dynamic scenes, which jointly reconstructs rigid and nonrigid parts without explicitly modelling the camera motion. Using dense correspondences, we derive a training objective that aims to opportunistically preserve pairwise distances between reconstructed 3D points. In this process, the dense depth map is learned implicitly using the as-rigid-as-possible hypothesis. Our method provides promising results, demonstrating its capability of reconstructing 3D from challenging videos of non-rigid scenes. Furthermore, the proposed method also provides unsupervised motion segmentation results as an auxiliary output.

Abstract. Airborne SAR on small and flexible platforms guarantees the evaluation of local damages after natural disasters and is both weather and daylight independent. The processing of circular flight trajectories can further improve the reconstruction of target scenes especially in complex urban scenarios as shadowing and foreshortening effects can be reduced by multiple views from different aspect angles (hyper- or full- aspect). A dataset collected with the Miranda 35 GHz radar system with 1 GHz bandwidth on a small ultralight aircraft on a circular trajectory over an urban scene was processed using a time domain approach. The SAR processing chain and the effects of the navigational data for such highly nonlinear trajectories and unstable platforms are described. The generated SAR image stack over the entire trajectory consists of 240 individual SAR images, each image visualizing the scene from a slightly different aspect angle. First results for the fusion of multiple aspect views to create one resulting image with reduced shadow areas and the possibility to find hidden targets are demonstrated. Further potentials of such particular datasets like moving target indication are discussed.

Given a collection of images of offices, what would we say we see in the images? The objects of interest are likely to be monitors, keyboards, phones, etc. Such identification of the foreground in a scene is important to avoid distractions caused by background clutter and facilitates better understanding of the scene. It is crucial for such an identification to be unsupervised to avoid extensive human labeling as well as biases induced by human intervention.Most interesting scenes contain multiple objects of interest. Hence, it would be useful to separate the foreground into the multiple objects it contains. We propose dISCOVER, an unsupervised approach to identifying the multiple objects of interest in a scene from a collection of images. In order to achieve this, it exploits the consistency in foreground objects - in terms of occurrence and geometry - across the multiple images of the scene.

Accurate reconstruction of dynamic objects in urban scenes remains a challenging task. Existing methods typically rely on complex 3D annotations to identify dynamic objects, which are expensive and difficult to obtain. In this paper, we propose a novel approach to address this challenge by leveraging accessible and effective 2D annotations. We utilize a 2D foundation model to identify dynamic objects within the image and lift this knowledge to 3D space, obtaining the 3D point cloud of dynamic objects. To capture the rigid motion patterns of dynamic objects, we introduce an object-level motion model together with the local aggregation strategy. Additionally, we propose a structural consistency constraint to maintain the consistency of dynamic objects during motion, significantly improving the accuracy of scene reconstruction. Extensive experiments demonstrate that our method achieves comparable reconstruction performance to methods relying on 3D annotations, providing a cost-effective and accurate solution for dynamic urban scene reconstruction.

This paper addresses the challenge that current event-based video reconstruction methods cannot produce static background information. Recent research has uncovered the potential of event cameras in capturing static scenes. Nonetheless, image quality deteriorates due to noise interference and detail loss, failing to provide reliable background information. We propose a two-stage reconstruction strategy to address these challenges and reconstruct static scene images comparable to frame cameras. Building on this, we introduce the URSEE framework designed for reconstructing motion videos with static backgrounds. This framework includes a parallel channel that can simultaneously process static and dynamic events, and a network module designed to reconstruct videos encompassing both static and dynamic scenes in an end-to-end manner. We also collect a real-captured dataset for static reconstruction, containing both indoor and outdoor scenes. Comparison results indicate that the proposed method achieves state-of-the-art performance on both synthetic and real data.

Criminator: An Easy-to-Use XR "Crime Animator" for Rapid Reconstruction and Analysis of Dynamic Crime Scenes

Video retrieval based on object discovery

Calibration is vital to improve robot accuracy. Automatic calibrations that require no extra devices have lots of conveniences, which is especially significant for cable-driven parallel robots that usually have the reconfigurable ability. This paper proposes a new automatic calibration method that is applicable for a general kind of redundantly actuated cable-driven parallel robots. The key point of this method is to establish the mapping between the unknown parameters to be calibrated and the parameters that could be measured by the inner sensors, and then use least squares algorithm to find the solutions. Specifically, the unknown parameters herein are the coordinates of the attachment points and the measured parameters are the lengths of the redundant cables. Simulations are performed on a 3-DOF parallel robot driven by 4 cables for validation. Results show the proposed calibration method could precisely find the real coordinate values of the attachment points, errors less than 10−12 mm. Trajectory simulations also indicate that the positioning accuracy of the CDPR could be greatly improved after calibration using the proposed method.

Semantic-aware dynamic scene tracking and reconstruction for mobile robots leveraging visual SLAM

3D Gaussian Splatting has demonstrated remarkable real-time rendering capabilities and superior visual quality in novel view synthesis for static scenes. Building upon these advantages, researchers have progressively extended 3D Gaussians to dynamic scene reconstruction. Deformation field-based methods have emerged as a promising approach among various techniques. These methods maintain 3D Gaussian attributes in a canonical field and employ the deformation field to transform this field across temporal sequences. Nevertheless, these approaches frequently encounter challenges such as suboptimal rendering speeds, significant dependence on initial point clouds, and vulnerability to local optima in dim scenes. To overcome these limitations, we present FRoG, an efficient and robust framework for high-quality dynamic scene reconstruction. FRoG integrates per-Gaussian embedding with a coarse-to-fine temporal embedding strategy, accelerating rendering through the early fusion of temporal embeddings. Moreover, to enhance robustness against sparse initializations, we introduce a novel depth- and error-guided sampling strategy. This strategy populates the canonical field with new 3D Gaussians at low-deviation initial positions, significantly reducing the optimization burden on the deformation field and improving detail reconstruction in both static and dynamic regions. Furthermore, by modulating opacity variations, we mitigate the local optima problem in dim scenes, improving color fidelity. Comprehensive experimental results validate that our method achieves accelerated rendering speeds while maintaining state-of-the-art visual quality.

While dynamic scene reconstruction has made revolutionary progress from the earliest setup using a mass of static cameras in studio environment to the latest egocentric or hand-held moving camera based schemes, it is still restricted by the recording volume, user comfortability, human labor and expertise. In this paper, a novel solution is proposed through a real-time and robust dynamic fusion scheme using a single flying depth camera, denoted as FlyFusion. By proposing a novel topology compactness strategy for effectively regularizing the complex topology changes, and the Geometry And Motion Energy (GAME) metric for guiding the viewpoint optimization in the volumetric space, FlyFusion succeeds to enable intelligent viewpoint selection based on the immediate dynamic reconstruction result. The merit of FlyFusion lies in its concurrent robustness, efficiency, and adaptation in producing fused and denoised 3D geometry and motions of a moving target interacting with different non-rigid objects in a large space.

We propose a new method for background modeling. Our method is based on the two complementary approaches. One uses the probability density function(PDF) to approximate background model. The PDF is estimated non-parametrically by using Parzen density estimation. And foreground object is detected based on the estimated PDF. The other method is based on the evaluation of the local texture at pixel-level resolution while reducing the effects of variations in lighting. Fusing their approach realize robust object detection under varying illumination. Several experiments show the effectiveness of our approach.

<div>The advancement in vision sensors and embedded technology created the opportunity in autonomous vehicles to look ahead in the future to avoid potential obstacles and steep regions to reach the target location as soon as possible and yet maintain vehicle safety from rollover. The present work focuses on developing a nonlinear model predictive controller (NMPC) for a high-speed off-road autonomous vehicle, which avoids undesirable conditions including stationary obstacles, moving obstacles, and steep regions while maintaining the vehicle safety from rollover. The NMPC controller is developed using CasADi tools in the MATLAB environment. The CasADi tool provides a platform to formulate the NMPC problem using symbolic expressions, which is an easy and efficient way of solving the optimization problem. In the present work, the vehicle lateral dynamics are modeled using the Pacejka nonlinear tire model. Further, a new algorithm is developed based on the box slope and box detection methods to process the stationary and moving obstacles. These methods use the vehicle’s current heading and generate a light detection and ranging (LIDAR) view through rectangular box regions. These box regions mimic the actual vision sensor regions, and logic can easily be applied to real vehicle conditions. From the results, it is observed that the vehicle avoids both moving and stationary lengthy obstacles and can safely navigate through a pool of obstacles that can mimic real-world off-road scenarios. Further, the simulations with Gaussian noise in vehicle state estimations and obstacle states confirmed that the developed algorithm can reach the target without collision by meeting all the vehicle safety constraints for the considered uncertainty limits.</div>

In this paper we describe a technique of fusing larger scenes from multiple views, using a landmark approach. For images of remote scenes, the fused image is reconstructed using sample mean shapes on affine shape manifolds. This manifold of configurations of points in general position is a real Grassmann manifold which can be equivariantly embedded in a space of matrices, and we consider the extrinsic means w.r.t. this embedding. On the other hand, for photographs taken from a close distance by a noncalibrated camera, the fused image is reconstructed using sample mean shapes on projective shape manifolds. This manifold of configurations of points under a given projective frame is a product of real projective planes and we consider extrinsic means w.r.t. to a standard embedding in a space of blocks of symmetric matrices. We illustrate these methods on two sets of images.