OptiNeRF: A Spatially Optimized Neural Rendering Framework for Complex Scene Reconstruction

A great challenge in 3D digitization and modelling lies in striking a balance between surface detail and model size, while accommodating the geometric information representation requirements of diverse archaeological practices. 3D meshes that contain a high polygon count severely impact storage requirements, data transmission, and 3D rendering. It is therefore typical for high-resolution digitized objects and modelled assets alike to be sub-sampled and simplified into levels of detail (LODs) to match the requirements of the intended applications. By doing so, however, geometric detail is lost, negatively affecting the model’s visual fidelity during 3D visualization. Normal mapping offers a reliable and practical solution to this challenge by transferring surface orientation data from the original, detailed models into simplified versions. This approach preserves the key visual characteristics present on the surface of a high-detail model, without reintroducing geometric complexity into the simplified structure. In archaeological contexts, where both accuracy and accessibility are important, normal mapping enables lightweight models to maintain the perceptual qualities of high-resolution originals, while supporting diverse visualization applications, from analytical inspection to public dissemination. In this paper, our proposed methodology explores the robustness of detail transfer through normal mapping on simplified archaeological models, highlighting its effectiveness as a tool for efficient cultural heritage visualization and immersive applications. The study claims that normal-mapped models can effectively stand in for the original high-detail scans in a variety of scenarios, thus offering a viable and efficient alternative to the demanding storage and transmission of high-fidelity 3D models.

Multi-focus image fusion is an effective way to overcome the shallow depth of field of ordinary cameras and to achieve all-in-focus imaging across the entire depth range. However, existing multi-focus image fusion methods are usually designed for single-view and static scenes, mainly because the current focal stack imaging technology for capturing multi-focus images can only capture one partially focused image at each moment. To simultaneously capture multi-focus images for all-in-focus imaging in dynamic scenes, we propose a computational imaging scheme for all-in-focus dynamic image fusion. Specifically, a multi-view multi-focus (MVMF) imaging technology that integrates the characteristics of both multi-view imaging and focal stack imaging is designed to simultaneously capture MVMF images. For the fusion of MVMF images, we design an MVMF image fusion method based on focus clarity evaluation. This method consists of three modules: image alignment, focus clarity evaluation, and image fusion, which can effectively fuse MVMF images into a high-quality all-in-focus image. In addition, to fill the gap of lacking dynamic multi-focus image datasets in the existing multi-focus image fusion field, we render an MVMF image dataset (i.e., DYN-MVMF) capturing dynamic scenes using the Blender software. On the DYN-MVMF dataset, we conduct extensive comparative experiments with state-of-the-art multi-focus image fusion methods. The experimental results validate the effectiveness of the proposed method in fusing MVMF images into a high-quality all-in-focus image. Moreover, we also conduct comparative experiments with the conventional focal stack imaging scheme on dynamic scene imaging and salient object detection tasks. The results demonstrate that our proposed computational imaging scheme can achieve all-in-focus imaging for each frame of dynamic scenes, thereby enabling more accurate salient object detection results. Therefore, the proposed scheme is expected to contribute to the development of high-performance imaging systems.

Agriculture is rapidly becoming a data-driven field where automation relies on transforming 2D images into accurate 3D models. However, selecting the most effective method remains challenging due to the unconstrained nature of the environment. This review assesses the effectiveness of geometry-based, sensor-based, and learning-based reconstruction methodologies in agricultural settings. We analyze photogrammetric pipelines, active sensing, and neural rendering methods based on their geometric accuracy, data processing speed, and field performance against wind or occlusion. Our analysis indicates that while Light Detection and Ranging (LiDAR) is highly accurate, it is too expensive for widespread adoption. Conversely, geometry-based methods are inexpensive but struggle with complex biological structures. Learning-based methods, especially 3D Gaussian Splatting (3DGS), have revolutionized the field by enabling a balance between visual fidelity and real-time inference speed. We conclude that the best chance for scalability and accuracy lies in hybrid pipelines that integrate Vision Foundation Models (VFMs) with geometric priors. We believe that "hybrid intelligence" systems, such as edge-native 3D Gaussian Splatting combined with semantic priors, are the future of 3D reconstruction. These systems will enable the creation of real-time, spatiotemporal (4D) digital twins that drive automated decision-making in precision agriculture.

In many cases, the Digital Surface Models (DSMs) and Digital Elevation Models (DEMs) are obtained with Light Detection and Ranging (LiDAR) or stereo matching. As an active method, LiDAR is very accurate but expensive, thus often limiting its use in small-scale acquisition. Stereo matching is suitable for large-scale acquisition of terrain information as the increase of satellite stereo sensors. However, underperformance of stereo matching easily occurs in textureless areas. Accordingly, this study proposed a Shading Aware DSM GEneration Method (SADGE) with high resolution multi-view satellite images. Considering the complementarity of stereo matching and Shape from Shading (SfS), SADGE combines the advantage of stereo matching and SfS technique. First, an improved Semi-Global Matching (SGM) technique is used to generate an initial surface expressed by a DSM; then, it is refined by optimizing the objective function which modeled the imaging process with the illumination, surface albedo, and normal object surface. Different from the existing shading-based DEM refinement or generation method, no information about the illumination or the viewing angle is needed while concave/convex ambiguity can be avoided as multi-view images are utilized. Experiments with ZiYuan-3 and GaoFen-7 images show that the proposed method can generate higher accuracy DSM (12.5–56.3% improvement) with sound overall shape and temporarily detailed surface compared with a software solution (SURE) for multi-view stereo.

Monocular depth reconstruction of complex and dynamic scenes is a highly challenging problem. While for rigid scenes learning-based methods have been offering promising results even in unsupervised cases, there exists little to no literature addressing the same for dynamic and deformable scenes. In this work, we present an unsupervised monocular framework for dense depth estimation of dynamic scenes, which jointly reconstructs rigid and nonrigid parts without explicitly modelling the camera motion. Using dense correspondences, we derive a training objective that aims to opportunistically preserve pairwise distances between reconstructed 3D points. In this process, the dense depth map is learned implicitly using the as-rigid-as-possible hypothesis. Our method provides promising results, demonstrating its capability of reconstructing 3D from challenging videos of non-rigid scenes. Furthermore, the proposed method also provides unsupervised motion segmentation results as an auxiliary output.

Moving vehicles segmentation based on Bayesian framework for Gaussian motion model

Rendering realistic illumination effects for complex animated scenes with many dynamic objects or characters is computationally expensive. Yet, it is not obvious how important such accurate lighting is for the overall perceived realism in these scenes. In this paper, we present a methodology to evaluate the perceived fidelity of illumination in scenes with dynamic aggregates, such as crowds, and explore several factors which may affect this perception. We focus in particular on evaluating how a popular spherical harmonics lighting method can be used to approximate realistic lighting of crowds. We conduct a series of psychophysical experiments to explore how a simple approach to approximating global illumination, using interpolation in the temporal domain, affects the perceived fidelity of dynamic scenes with high geometric, motion, and illumination complexity. We show that the complexity of the geometry and temporal properties of the crowd entities, the motion of the aggregate as a whole, the type of interpolation (i.e., of the direct and/or indirect illumination coefficients), and the presence or absence of colour all affect perceived fidelity. We show that high (i.e., above 75%) levels of perceived scene fidelity can be maintained while interpolating indirect illumination for intervals of up to 30 frames, resulting in a greater than three‐fold rendering speed‐up.

While 3D GANs have recently demonstrated the high-quality synthesis of multi-view consistent images and 3D shapes, they are mainly restricted to photo-realistic human portraits. This paper aims to extend 3D GANs to a different, but meaningful visual form: artistic portrait drawings. However, extending existing 3D GANs to drawings is challenging due to the inevitable geometric ambiguity present in drawings. To tackle this, we present Dr.3D, a novel adaptation approach that adapts an existing 3D GAN to artistic drawings. Dr.3D is equipped with three novel components to handle the geometric ambiguity: a deformation-aware 3D synthesis network, an alternating adaptation of pose estimation and image synthesis, and geometric priors. Experiments show that our approach can successfully adapt 3D GANs to drawings and enable multi-view consistent semantic editing of drawings.

Recent advancements in 4D scene reconstruction using neural radiance fields (NeRF) have demonstrated the ability to represent dynamic scenes from multi-view videos. However, they fail to reconstruct the dynamic scenes and struggle to fit even the training views in unsynchronized settings. It happens because they employ a single latent embedding for a frame while the multi-view images at the same frame were actually captured at different moments. To address this limitation, we introduce time offsets for individual unsynchronized videos and jointly optimize the offsets with NeRF. By design, our method is applicable for various baselines and improves them with large margins. Furthermore, finding the offsets always works as synchronizing the videos without manual effort. Experiments are conducted on the common Plenoptic Video Dataset and a newly built Unsynchronized Dynamic Blender Dataset to verify the performance of our method. Project page: https://seoha-kim.github.io/sync-nerf

• Design a self-attention aggregation module using a pre-trained diffusion model. • Integrate high-precision masks into RGB-D SLAM for robust dynamic tracking. • Validate superior accuracy and efficiency on dynamic simulation datasets. In dynamic environments, performing RGB-D SLAM (Simultaneous Localization and Mapping) faces significant challenges primarily due to the presence of moving objects. The motion of these objects can introduce tracking errors and inaccuracies in map construction, thereby compromising the stability and overall performance of the system. To maintain high-precision localization and mapping under such conditions, a SLAM system must effectively detect and handle dynamic objects. To address these challenges, this paper presents a novel RGB-D SLAM method, referred to as DMSAA-SLAM (Dynamic Scene SLAM Based on Diffusion Model Self-Attention Aggregation). The core idea is to leverage a pre-trained stable diffusion model, particularly its self-attention layers, to handle the complexity of dynamic scenes. By employing a multi-resolution aggregation approach, combined with iterative merging and nonmaximum suppression, the proposed method generates high-precision segmentation masks. These masks enable fine-grained segmentation of moving objects and effectively eliminate dynamic feature points, thereby mitigating the impact of dynamic elements on the SLAM process and ensuring efficient and accurate tracking and mapping.

Novel view synthesis of remote sensing scenes is of great significance for scene visualization, human-computer interaction, and various downstream applications. Despite the recent advances in computer graphics and photogrammetry technology, generating novel views is still challenging particularly for remote sensing images due to its high complexity, view sparsity and limited view-perspective variations. In this paper, we propose a novel remote sensing view synthesis method by leveraging the recent advances in implicit neural representations. Considering the overhead and far depth imaging of remote sensing images, we represent the 3D space by combining implicit multiplane images (MPI) representation and deep neural networks. The 3D scene is reconstructed under a self-supervised optimization paradigm through a differentiable multiplane renderer with multi-view input constraints. Images from any novel views thus can be freely rendered on the basis of the reconstructed model. As a by-product, the depth maps corresponding to the given viewpoint can be generated along with the rendering output. We refer to our method as Implicit Multiplane Images (ImMPI). To further improve the view synthesis under sparse-view inputs, we explore the learning-based initialization of remote sensing 3D scenes and proposed a neural network based Prior extractor to accelerate the optimization process. In addition, we propose a new dataset for remote sensing novel view synthesis with multi-view real-world google earth images. Extensive experiments demonstrate the superiority of the ImMPI over previous state-of-the-art methods in terms of reconstruction accuracy, visual fidelity, and time efficiency. Ablation experiments also suggest the effectiveness of our methodology design. Our dataset and code can be found at https://github.com/wyc-Chang/ImMPI

This paper addresses the problem of fully automated mining of public space video data, a highly desirable capability under contemporary commercial and security considerations. This task is especially challenging due to the complexity of the object behaviors to be profiled, the difficulty of analysis under the visual occlusions and ambiguities common in public space video, and the computational challenge of doing so in real-time. We address these issues by introducing a new dynamic topic model, termed a Markov Clustering Topic Model (MCTM). The MCTM builds on existing dynamic Bayesian network models and Bayesian topic models, and overcomes their drawbacks on sensitivity, robustness and efficiency. Specifically, our model profiles complex dynamic scenes by robustly clustering visual events into activities and these activities into global behaviours with temporal dynamics. A Gibbs sampler is derived for offline learning with unlabeled training data and a new approximation to online Bayesian inference is formulated to enable dynamic scene understanding and behaviour mining in new video data online in real-time. The strength of this model is demonstrated by unsupervised learning of dynamic scene models for four complex and crowded public scenes, and successful mining of behaviors and detection of salient events in each.

In this paper, we address the problem of referring expression comprehension in videos, which is challenging due to complex expression and scene dynamics. Unlike previous methods which solve the problem in multiple stages (i.e., tracking, proposal-based matching), we tackle the problem from a novel perspective, co-grounding, with an elegant one-stage framework. We enhance the single-frame grounding accuracy by semantic attention learning and improve the cross-frame grounding consistency with co-grounding feature learning. Semantic attention learning explicitly parses referring cues in different attributes to reduce the ambiguity in the complex expression. Co-grounding feature learning boosts visual feature representations by integrating temporal correlation to reduce the ambiguity caused by scene dynamics. Experiment results demonstrate the superiority of our framework on the video grounding datasets VID and LiOTB in generating accurate and stable results across frames. Our model is also applicable to referring expression comprehension in images, illustrated by the improved performance on the RefCOCO dataset. Our project is available at https://sijiesong.github.io/co-grounding.

The goal of this paper is to improve the efficiency of collision detection algorithms applied to highly complex geometry scenes not having geometry coherence. Efficiency is improved by reducing the number of objects to be checked for collisions and accelerating the fundamental repeating checks on which the collision detection process relies. The number of objects to be checked is decreased by combining hierarchical representation techniques with incremental methods. The fundamental repeating check, actually a collision check between two bounding boxes, is accelerated by using 3D bounding boxes each having an associated transformation for mapping its vertices from a local to a world coordinate system (CS). This acceleration is based on the observation that two boxes (A andB) intersect if and only if the projections ofA andB intersect on all three axis-aligned orthographic views, both forA in the local CS ofB and forB in the local CS ofA. These algorithms have been successfully implemented in simulating dynamic scenes with high geometry complexity and as avirtual collision detection sensor for off-line robot motion planning.

The Drosophila visual system is extremely sensitive to moving targets, which provides a wealth of biological inspiration for the research of target motion perception in complex scenes, and also lays a biological theoretical foundation for the establishment of artificial drosophila visual neural networks. Drosophila's vision has been extensively studied in physiology, anatomy, and behavior, but our understanding of its underlying neural computing is still insufficient. In order to gain insight into the neural mechanism in Drosophila vision and take better advantage of its superiority in motion perception, we propose a Drosophila vision-inspired model, which constructs a complete Drosophila visual motion perception system by integrating continuous computing layers. Our hybrid model can fully demonstrate the motion perception process in Drosophila vision. In addition, the Drosophila vision-inspired model can also be exploited to salient object detection in dynamic scenes. This novel salient object detection model is different from the previous in that it can accurately identify the motion of interest (MOI) while suppressing background disturbances and ego-motion. Comprehensive evaluations using standard benchmarks demonstrate the superiority of our model in salient object detection compared with the state-of-the-art methods.