Depth-Regularized 3D Gaussian Splatting for Indoor Panoramic images

Panoramic sensing system has attracted great attention in intelligent vehicle because of the wide range of perception areas. In order to make the spatial direction of each pixel more precise in panoramic image, we propose a panoramic image mosaics assisted by LiDAR. Firstly, a fusion system consisting of LiDAR and panoramic camera is built, and the related calibration equipment is elaborately designed for the system considering the sparse point cloud from 16-line LiDAR. Secondly, the center point of calibration plate is utilized to construct the geometry constraint for the extrinsic calibration of dual fisheye cameras, and the rotation matrix of calibration parameters makes the alignment of dual fisheye images. Thirdly, in order to offset calibration deviation and the distance between optical centers, the cos(x) smoothing function is introduced to eliminate the splicing marks in final panoramic image. Experimental results show that the proposed method can effectively be used in panoramic image mosaics, and the spatial direction of each pixel in the panoramic image is accurate.

In this paper, a new panoramic image generation algorithm is proposed based on more realistic image formation processes. Perspective projection, lens distortion, vignetting and illumination effects are incorporated into the proposed panoramic modeling. Intrinsic and extrinsic camera parameters are estimated by the proposed stable camera parameter estimation algorithm derived from panning camera constraints. This paper shows that accurate panoramic images can be reconstructed based on the proposed camera modeling and parameters estimation. The effectiveness of the proposed algorithm is also shown with several image sequences in terms of reconstruction error from the generated panoramic image.

As a powerful representation of 3D scenes, the neural radiance field (NeRF) enables high-quality novel view synthesis from multi-view images. Stylizing NeRF, however, remains challenging, especially in simulating a text-guided style with both the appearance and the geometry altered simultaneously. In this paper, we present NeRF-Art, a text-guided NeRF stylization approach that manipulates the style of a pre-trained NeRF model with a simple text prompt. Unlike previous approaches that either lack sufficient geometry deformations and texture details or require meshes to guide the stylization, our method can shift a 3D scene to the target style characterized by desired geometry and appearance variations without any mesh guidance. This is achieved by introducing a novel global-local contrastive learning strategy, combined with the directional constraint to simultaneously control both the trajectory and the strength of the target style. Moreover, we adopt a weight regularization method to effectively suppress cloudy artifacts and geometry noises which arise easily when the density field is transformed during geometry stylization. Through extensive experiments on various styles, we demonstrate that our method is effective and robust regarding both single-view stylization quality and cross-view consistency.

Image stitching is an attractive method to merging multiple images. It can produce a wide-angle panoramic photograph while maintaining the quality of the source images. The process is simply performed by overlapping part of the images which contain common scene. Today, panoramic image stitching is widely used in applications such as 360-degree cameras and virtual reality photography. If the stitching technology ensure fast processing speed and accurate performance, it is expected to produce diverse image contents in real time. Moreover, the panoramic images can be mapped to various projective layouts, e.g. spherical and stereographic projection. The mapping of the panoramic image along the suitable projective layout makes it possible to express more stereoscopic space, that is, the space in the photo can give the impression that it looks as if it is in front. It can be utilized when creating 3D reconstruction space with high-resolution panoramic images. This paper describes the concept of feature based image stitching and presents the implementation results. It also provides the results of stitching with more than 10 images and suggests the time-efficient way for stitching multiple images. The time-efficient stitching method can be applied to various applications so that panoramic images can be easily accessed in everyday life.

Automatic detection and locating of objects such as poles, traffic signs, and building corners in street scenes captured from a mobile mapping system has many applications. Template matching is a technique that could automatically recognise the counterparts or correspondents of an object from multi-view images. In this study, we aim at finding correspondents of an object from wide baseline panoramic images with large geometric deformations from sphere projection and significant systematic errors from multi-camera rig geometry. Firstly, we deduce the camera model and epipolar model of a multi-camera rig system. Then, epipolar errors are analysed to determine the search area for pixelwise matching. A low-cost laser scanner is optionally used to constrain the depth of an object. Lastly, several classic feature descriptors are introduced to template matching and evaluated on the multi-view panoramic image dataset. We propose a template matching method combining a fast variation of a scale-invariant feature transform (SIFT) descriptor. Our method experimentally achieved the best performance in terms of accuracy and efficiency comparing to other feature descriptors and the most recent robust template matching methods.

Abstract. In this paper, we propose the 3D surface reconstruction scheme using multi-view and multi-date Google Earth (GE) satellite images. Multi-view stereo matching (MVS) scheme is one of the methods for reconstructing dense 3D surface based on multi-view images and corresponding camera pose geometry. If many views are input, MVS can estimate the disparity (depth) by matching pixels. However, the common users are not always possible to obtain both multi-view satellite images and the camera geometry (such as Rational Polynomial Camera) in various earth regions. Instead, the GE provides multi-view and multi-date satellite images of earth regions. Therefore, the goal of the proposed method is to perform a 3D surface reconstruction using the GE satellite image. We suppose that the GE satellite image is a pinhole camera model, and the camera pose geometry is estimated using the perspective projection model (PPM) based structure from the motion (SfM) method. Then the 3D surface is reconstructed and fusion using the MVS method. However, the GE satellite image is a transformed pseudo-orthoimage for integration into the raster image. For this reason, the camera pose geometry is inaccurately computed in the SfM process. Thus, the high-rise structures in the reconstructed 3D surface are distorted (distorted hexahedral 3D space). Importantly, the satellite image is a weak PPM and it can express the orthographic projection model. Therefore, we compute 3D homography for transforming between distorted hexahedral space to orthographic cuboid space. Then, the distorted 3D surface is transformed using a projective reconstruction based on 3D homography. The transformed 3D surface has the correct shape in the orthographic projection model. The advantage of the proposed method is that the 3D surface of various earth regions is reconstructed using simply accessible GE satellite images. And the transformed 3D surface is reconstructed into orthographic projection model space, thus the orthoimage can be generated using projection.

With the increased availability of multi-view satellite images, the number of investigations on 3D urban scene reconstruction from multiple satellite images is also increasing. Conventional Multi-View Stereo (MVS) pipelines require the calibrated pose information of the satellite cameras to determine the epipolar geometry and the 3D structure of the stereo correspondences. In this study, we propose a novel Monocular Height estimation and Fusion (MHF) method for 3D reconstruction from uncalibrated multi-view satellite images. By employing a learned monocular depth network, the proposed method first obtains the height map of each satellite image. Second, all height maps obtained from the multi-view images are fused to a refined height map in each image plane. To fuse the height maps, all maps are affine transformed to a virtual reference coordinate system and the transformed maps are then projected to the image plane of each camera coordinate system. The monocular depth network was trained and evaluated on the Data Fusion Contest 2019 (DFC19) dataset including Jacksonville, FL, and Omaha, NE. We also evaluate the ATL-SN4 dataset covering Atlanta, GA to test on untrained new urban scenes.

With the increasing popularity of wind turbines, the demand for integrity detecting of wind turbines operating in natural or extreme environments is also increasing. To better detect the state of wind turbines, rendering and 3D reconstruction of realistic wind turbine models has become a crucial task. The neural radiation field is becoming a widely used method for novel view synthesis and 3D reconstruction. In the original neural radiation field method, the scene or object is required to have a large number of features or complex textures. But for wind turbines, the surface is smooth and texture-free, which creates blur and ghosting in the results. Therefore, we propose the Wind Turbine Neural Radiance Fields (WTBNeRF), a network dedicated to wind turbine rendering and 3D reconstruction. Instead of single pixel-centered rays, we use conical truncated rays to cover individual pixel ranges in greater detail, effectively reducing aliasing and blurring in smooth, low-texture wind turbine scenes. At the same time, obtaining accurate camera poses for low-texture objects and scenes is also a challenging task. We use a pretrained camera pose estimation neural radiation field network to predict the camera pose of the wind turbines in the dataset, reducing the requirement to know the real camera parameters of the data in advance. Moreover, in the network design, we simplify the network structure, which greatly reduced the network training time. The speed is about 10 times faster than that of NeRF for our multi-scale wind turbine dataset.KeywordsWind turbine blade3D ReconstructionNovel view synthesis

As core infrastructure of power transmission networks, power towers require high-precision 3D models, which are critical for intelligent inspection and digital twin applications of power transmission lines. Traditional reconstruction methods, such as LiDAR scanning and oblique photogrammetry, suffer from issues including high operational risks, low modeling efficiency, and loss of fine details. To address these limitations, this paper proposes a 3D Gaussian Splatting (3DGS)-based method for power tower 3D reconstruction to enhance reconstruction efficiency and detail preservation capability. First, a multi-view data acquisition scheme combining “unmanned aerial vehicle + oblique photogrammetry” was designed to capture RGB images acquired by Unmanned Aerial Vehicle (UAV) platforms, which are used as the primary input for 3D reconstruction. Second, a sparse point cloud was generated via Structure from Motion. Finally, based on 3DGS, Gaussian model initialization, differentiable rendering, and adaptive density control were performed to produce high-precision 3D models of power towers. Taking two typical power tower types as experimental subjects, comparisons were made with the oblique photogrammetry + ContextCapture method. Experimental results demonstrate that 3DGS not only achieves high model completeness (with the reconstructed model nearly indistinguishable from the original images) but also excels in preserving fine details such as angle steels and cables. Additionally, the final modeling time is reduced by over 70% compared to traditional oblique photogrammetry. 3DGS enables efficient and high-precision reconstruction of power tower 3D models, providing a reliable technical foundation for digital twin applications in power transmission lines. By significantly improving reconstruction efficiency and reducing operational costs, the proposed method supports sustainable power infrastructure inspection, asset lifecycle management, and energy-efficient digital twin applications.

Multi-view satellite images can form multiple stereo image pairs (MSIPs). Each pair allows generating two types of surface information: surface disparity maps (SDMs) and digital elevation models (DEMs). Both types are important for 3D assisted information extraction applications. Since multi-view images can be used to generate more accurate stereo-based DEMs than using one stereo-pair, it is more computationally cost-effective to generate enhanced SDMs from MSIPs. However, the disparity values from different stereo-pairs are unequal. Hence, redundant data cannot be generated to enhance the SDMs. Therefore, our research focuses on the generation of inter-proportional disparity maps from multi-view very high resolution satellite images. The method uses the sensor model information of the stereo images to construct multiple epipolar images in the object-space. After implementation, the disparity values of the constructed images were found to be proportional. This property allows the generation of SDMs that may replace the DEMs to some extent.

At present, the widely used traditional three-dimensional (3D) reconstruction techniques are still insufficient to adapt to various diverse scenarios. Compared to traditional methods, emerging technologies like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) for novel view synthesis offer more realistic and comprehensive expression capabilities. However, most of these related technologies still rely on traditional methods and require extensive and dense input views, which poses challenges to the reconstruction in the real-world scenarios. We propose MFGaussian, a framework based on 3DGS for 3D scene representation by fusing multi-modal data obtained from the Mobile laser scanning system (MLS) to achieve high robustness and accuracy even with limited input views. MFGaussian employs stepwise training approach to independently learn the global information and details of the scene. During pre-training, a substantial number of virtual training views are generated by projecting color point clouds, thereby enhancing the model's robustness. Subsequently, the model is fine-tuned using the original training views. This method initializes the laser point cloud as 3D Gaussian, obtains camera parameters through multi-sensor calibration and subsequent spherical interpolation, thus obtaining high-precision initial data without relying on Structure from Motion(SfM), and further ensures accurate geometric structure through the partial optimization. Furthermore, an analysis has been conducted on how variations in lighting brightness within the scene affect the view synthesis from diverse perspectives and positions, with an appearance model incorporated to eliminate the resulting color ambiguity. Our method, tested on our dataset and the ETH3D stereo benchmark, demonstrates enhanced capability and robustness of 3DGS in diverse scenarios without SfM or dense view inputs. It outperforms several state-of-the-art methods in both quantitative and qualitative evaluations. Our code will be open sourced soon later after the publication of this manuscript (https://github.com/oucliuyang/MFGaussian).

We present a “3D environment detection using multi-view color images and LiDAR point clouds” in this paper. Combining “multi-view color images” and “LiDAR point clouds” to solve the problem of insufficient resolution of traditional LiDAR point clouds to further realize accurate three-dimensional environment detection and recognition. The system adopts eight cameras to capture multi-view panorama images and combines with the depth information of the LiDAR point clouds to establish high-resolution 3D color images and depth images for subsequent objects detection and segmentation. According to the 3D point clouds information and multi-view images, 3D dynamic environment preprocessing, data processing and data format conversion are performed, the neural network input signal is extracted. Then through the neural network training, to determine the three-dimensional environment, and cutting out obstacles such as cars and pedestrians.

Improving deep learning based segmentation of scars using multi-view images

The success of neural fields for 3D vision tasks is now indisputable. Following this trend, several methods aiming for visual localization (e.g., SLAM) have been proposed to estimate distance or density fields using neural fields. However, it is difficult to achieve high localization performance by only density fields-based methods such as Neural Radiance Field (NeRF) since they do not provide density gradient in most empty regions. On the other hand, distance field-based methods such as Neural Implicit Surface (NeuS) have limitations in objects’ surface shapes. This paper proposes Neural Distance-Density Field (NeDDF), a novel 3D representation that reciprocally constrains the distance and density fields. We extend distance field formulation to shapes with no explicit boundary surface, such as fur or smoke, which enable explicit conversion from distance field to density field. Consistent distance and density fields realized by explicit conversion enable both robustness to initial values and high-quality registration. Furthermore, the consistency between fields allows fast convergence from sparse point clouds. Experiments show that NeDDF can achieve high localization performance while providing comparable results to NeRF on novel view synthesis. The code is available at https://github.com/ueda0319/neddf.

We present a method which enables rapid and dense reconstruction of scenes browsed by a single live camera. We take point-based real-time structure from motion (SFM) as our starting point, generating accurate 3D camera pose estimates and a sparse point cloud. Our main novel contribution is to use an approximate but smooth base mesh generated from the SFM to predict the view at a bundle of poses around automatically selected reference frames spanning the scene, and then warp the base mesh into highly accurate depth maps based on view-predictive optical flow and a constrained scene flow update. The quality of the resulting depth maps means that a convincing global scene model can be obtained simply by placing them side by side and removing overlapping regions. We show that a cluttered indoor environment can be reconstructed from a live hand-held camera in a few seconds, with all processing performed by current desktop hardware. Real-time monocular dense reconstruction opens up many application areas, and we demonstrate both real-time novel view synthesis and advanced augmented reality where augmentations interact physically with the 3D scene and are correctly clipped by occlusions.