Remote Sensing Novel View Synthesis With Implicit Multiplane Representations

The recent advances in computer graphics has spurred interest from both academics and industries in virtual reality (VR) enabled training applications. This paper presents a desktop VR prototype for industrial training applications. It is designed and implemented as a general shell by providing the data interface to import both the virtual environment models and specific domain knowledge. The geometric models of the virtual environment are constructed using feature-based modelling and assembly function by external CAD tools, and then transferred into the prototype through a conversion module. A hierarchical structure is proposed to partition and organise these imported virtual environment models. Based on this structure, a visibility culling approach is developed for fast rendering and user interaction. The case study has demonstrated the functionality of the proposed prototype system by applying it to a maintenance training application for a refinery bump system, which, in general, has a large number of polygons and a certain depth complexity. Significant speedup in both context rendering and response to user manipulations has been achieved to provide the user with a fast system response within the desktop virtual environment. Compared with the immersive VR system, the proposed system has offered an affordable and portable training media for industrial applications.

This talk will concentrate on how some of the recent advances in computer graphics and vision fit into a visual effects pipeline and their application for digitizing props. The production of movies often requires digitizing physical properties of various props. At Weta Digital we model, texture, and shade thousands of props in immense detail. We present a vision-based pipeline for digitizing props that requires far less user input than previous methods. This vision based pipeline fully automates the process of creating virtual cameras, reference geometry, and texture maps, which is then used by our artists to create the final assets at a much faster rate.

Recent advances in computer graphics and interactive techniques have increased the visual quality and flexibility of Augmented Reality (AR) applications. This, in turn has increased the viability of applying AR to educational exhibits for use in Science Centres, Museums, Libraries and other education centres. This article outlines a selection of five projects developed at the Human Interface Technology Laboratory in New Zealand (HIT Lab NZ) that have explored different techniques for applying AR to educational exhibits.These exhibits have received very positive feedback and appear to have educational benefits involving spatial, temporal and contextual conceptualisation and provide kinaesthetic, explorative and knowledge-challenging stimulus. The controls available to a user of turning a page, moving an AR marker, moving their head and moving a slider provide sufficient freedom to create many interaction scenarios that can serve educative outcomes. While the use of virtual media provides many advantages, creating new content is still quite difficult, requiring specialist software and skills. Useability observations are shared.

In this paper, we propose MINE to perform novel view synthesis and depth estimation via dense 3D reconstruction from a single image. Our approach is a continuous depth generalization of the Multiplane Images (MPI) by introducing the NEural radiance fields (NeRF). Given a single image as input, MINE predicts a 4-channel image (RGB and volume density) at arbitrary depth values to jointly reconstruct the camera frustum and fill in occluded contents. The reconstructed and inpainted frustum can then be easily rendered into novel RGB or depth views using differentiable rendering. Extensive experiments on RealEstate10K, KITTI and Flowers Light Fields show that our MINE outperforms state-of-the-art by a large margin in novel view synthesis. We also achieve competitive results in depth estimation on iBims-1 and NYU-v2 without annotated depth supervision. Our source code is available at https://github.com/vincentfung13/MINE.

Novel View Synthesis (NVS) is an important task for 3D interpretation in remote sensing scenes, which also benefits vicinagearth security by enhancing situational awareness capabilities. Recently, NVS methods based on Neural Radiance Fields (NeRFs) have attracted increasing attention for self-supervised training and highly photo-realistic synthesis results. However, it is still challenging to synthesize novel view images in remote sensing scenes, given the complexity of land covers and the sparsity of input multi-view images. In this paper, we propose a novel NVS method named FReSNeRF, which combines Image-Based Rendering (IBR) and NeRF to achieve high-quality results in remote sensing scenes with sparse input. We effectively solve the degradation problem by adopting the sampling space annealing method. Additionally, we introduce depth smoothness based on the segmentation mask to constrain the scene geometry. Experiments on multiple scenes show the superiority of our proposed FReSNeRF over other methods.

Storing and processing Remote Sensing (RS) images require large amounts of memory space and computing resources. Consequently, RS images are compressed and stored in various compression formats, such as JPEG2000. However, the processing of RS images for machine interpretation and understanding still necessitates the deployment of an image decompression stage in its entirety, followed by a computationally demanding image analysis pipeline. The image analysis stage is commonly composed of machine learning techniques, such as Deep Convolutional Neural Network (DCNN) models. Classification of remote sensing images is among the most common image analysis tasks. In the scope of this paper, we propose a sub-band image based classification method for the Remote Sensing Scene Classification (RSSC) task in the JPEG2000 compressed domain. The proposed approach exploits the already available sub-band image coefficients to classify RS images without needing for full decompression. Our study shows that our method increases the high frequency information in the LL sub-band and allows the image to contain more detail, leading to improved classifier performance while taking advantage of the partial decompression method.

Novel view synthesis is an important problem in computer vision and graphics. Over the years a large number of solutions have been put forward to solve the problem. However, the large-baseline novel view synthesis problem is far from being solved. Recent works have attempted to use Convolutional Neural Networks (CNNs) to solve view synthesis tasks. Due to the difficulty of learning scene geometry and interpreting camera motion, CNNs are often unable to generate realistic novel views. In this paper, we present a novel view synthesis approach based on stereo-vision and CNNs that decomposes the problem into two sub-tasks: view dependent geometry estimation and texture inpainting. Both tasks are structured prediction problems that could be effectively learned with CNNs. Experiments on the KITTI Odometry dataset show that our approach is more accurate and significantly faster than the current state-of-the-art. The code and supplementary material will be publicly available. Results could be found here this https URL

Novel View Synthesis (NVS) seeks to generate realistic novel views from limited source images, offering an effective solution for 3D reconstruction in complex or unknown environments. Achieving high generalization under occlusion, varying illumination, and sparse observations remains challenging, largely hinging on the effective extraction, optimization, and fusion of image features and spatial geometry. In this work, we propose JOIG — a Joint Optimization Model of Image Features and Constraint Geometry Fusion for generalizable 3D Gaussian splatting. JOIG introduces three key components: Multiscale Dimension Rotation Fusion (MDRF) to capture intrinsic dependencies across feature dimensions for enhanced image encoding, Geometry Self-Correcting Aggregation (GSCA) to refine multi-view geometry with depth-guided reweighting, and Geometry-Image Feature Aggregation (GIFA) to achieve pixel-aligned fusion of spatial and image information. Extensive experiments on DTU, LLFF, NeRF Synthetic, and Tanks and Temples datasets demonstrate that JOIG achieves state-of-the-art generalization performance, significantly improving both quantitative metrics and visual fidelity in novel view synthesis.

In recent decades, it is easy to obtain remote sensing images which have been successfully applied to various applications, such as urban planning, hazard monitoring, etc. In particular, high resolution (HR) remote sensing (RS) images can better monitor our living environment from a broader spatial perspective. However, raw remote sensing images provide no labeling information to train a classifier, which usually is exploited to generate remote sensing maps. Based on our previous work, in the paper, an automatic classification system is proposed to classify high resolution urban RS images using deep neural networks, in particular, convolutional neural networks and fully convolutional networks. The labeling information is assigned on the context of both social media photos and HR remote sensing images by significantly reducing the cost of manual labeling without the necessity of remote sensing experts. The experiments carried out on high resolution remote sensing images acquired in the city Frankfurt taken by the Jilin-1 satellites confirm the effectiveness of the proposed strategy compared to the state of the art.

Radiance field based methods have recently revolutionized novel view synthesis of scenes captured with multi-view photos. A significant recent advance is 3D Gaussian Splatting (3DGS), which utilizes a set of 3D Gaussians to represent a radiance field, yielding high-fidelity results in real-time rendering. However, we have observed that 3DGS struggles to capture the necessary details in sparsely observed regions, where there is not enough gradient for effective split and clone operations. In this paper, we present a novel solution to address this limitation. Our key idea is to leverage segmentation information to identify poorly optimized regions within the 3D Gaussian representation. By applying split or clone operations on the corresponding 3D Gaussians in these regions, we aim to refine the spatial distribution of Gaussians and enhance the overall quality of high-fidelity 3D scene reconstruction. To further optimize the reconstruction process, we introduce two spatial regularization terms: repulsion loss and smoothness loss. These terms effectively minimize overlap and redundancy among Gaussians, reducing outliers in the synthesized geometry. By incorporating these regularization techniques, our approach achieves state-of-the-art performance in real-time novel view synthesis and significantly improves visibility in less observed regions, leading to a more compact and accurate 3D scene representation.

Abstract. With deepening application of deep learning technology in the field of remote sensing, several challenges persist in the segmentation of remote sensing optical images. These challenges include: (1) insufficient availability of deep learning-based remote sensing semantic segmentation datasets; (2) inadequate utilization of multi-source remote sensing data in the field of semantic segmentation; (3) limited sample size for effective model training, as well as the need to enhance both the speed and accuracy of model training. To address these challenges, this study introduces a multi-source remote sensing dataset consisting of 15000 data pairs, each comprising remote sensing multispectral data, synthetic aperture radar(SAR) images, land use and land cover(LULC) data, digital elevation model (DEM) data as well as the analysis data including Slope, Aspect, and Hillshade. Through the application of an end-to-end network based on pix2pix, effective fusion and feature enhancement of multi-source remote sensing data were achieved. The structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), and Spectral Angle Mapper(SAM) values reached 0.84, 23.14, and 0.19, respectively, representing significant improvements over the baseline Pix2pix model’s performance metrics of 0.62, 18.84, and 0.23. In downstream semantic segmentation applications, the enhanced dataset was utilized to train semantic segmentation models for remote sensing image analysis. This approach effectively improved the training speed and segmentation accuracy of the models, with the mean intersection over union (mIoU) increasing from 0.467 to 0.481 and accuracy rising from 0.734 to 0.746. Moreover, the visual representation of remote sensing image segmentation demonstrated noticeable enhancements.

Timely acquiring the earthquake-induced damage of buildings is crucial for emergency assessment and post-disaster rescue. Optical remote sensing is a typical method for obtaining seismic data due to its wide coverage and fast response speed. Convolutional neural networks (CNNs) are widely applied for remote sensing image recognition. However, insufficient extraction and expression ability of global correlations between local image patches limit the performance of dense building segmentation. This paper proposes an improved Swin Transformer to segment dense urban buildings from remote sensing images with complex backgrounds. The original Swin Transformer is used as a backbone of the encoder, and a convolutional block attention module is employed in the linear embedding and patch merging stages to focus on significant features. Hierarchical feature maps are then fused to strengthen the feature extraction process and fed into the UPerNet (as the decoder) to obtain the final segmentation map. Collapsed and non-collapsed buildings are labeled from remote sensing images of the Yushu and Beichuan earthquakes. Data augmentations of horizontal and vertical flipping, brightness adjustment, uniform fogging, and non-uniform fogging are performed to simulate actual situations. The effectiveness and superiority of the proposed method over the original Swin Transformer and several mature CNN-based segmentation models are validated by ablation experiments and comparative studies. The results show that the mean intersection-over-union of the improved Swin Transformer reaches 88.53%, achieving an improvement of 1.3% compared to the original model. The stability, robustness, and generalization ability of dense building recognition under complex weather disturbances are also validated.

Target detection in remote sensing images has garnered significant attention due to its wide range of applications. Many traditional methods primarily rely on unimodal data, which often struggle to address the complexities of remote sensing environments. Furthermore, small-target detection remains a critical challenge in remote sensing image analysis, as small targets occupy only a few pixels, making feature extraction difficult and prone to errors. To address these challenges, this paper revisits the existing multimodal fusion methodologies and proposes a novel framework of separation before fusion (SBF). Leveraging this framework, we present Sep-Fusion—an efficient target detection approach tailored for multimodal remote sensing aerial imagery. Within the modality separation module (MSM), the method separates the three RGB channels of visible light images into independent modalities aligned with infrared image channels. Each channel undergoes independent feature extraction through the unimodal block (UB) to effectively capture modality-specific features. The extracted features are then fused using the feature attention fusion (FAF) module, which integrates channel attention and spatial attention mechanisms to enhance multimodal feature interaction. To improve the detection of small targets, an image regeneration module is exploited during the training stage. It incorporates the super-resolution strategy with attention mechanisms to further optimize high-resolution feature representations for subsequent positioning and detection. Sep-Fusion is currently developed on the YOLO series to make itself a potential real-time detector. Its lightweight architecture enables the model to achieve high computational efficiency while maintaining the desired detection accuracy. Experimental results on the multimodal VEDAI dataset show that Sep-Fusion achieves 77.9% mAP50, surpassing many state-of-the-art models. Ablation experiments further illustrate the respective contribution of modality separation and attention fusion. The adaptation of our multimodal method to unimodal target detection is also verified on NWPU VHR-10 and DIOR datasets, which proves Sep-Fusion to be a suitable alternative to current detectors in various remote sensing scenarios.

The use of deep neural networks for semantic segmentation of RS (remote sensing) images is an important research topic in the field of remote sensing intelligent interpretation, which has significant application value in urban planning, disaster assessment, carbon estimation, and other fields. However, high-resolution RS images bring problems of large data scale, high computational complexity, diverse scales, and irregular shapes. Moreover, there may be distant spatial relationships between target objects, requiring the utilization of long-range contextual information for accurate semantic segmentation. Inspired by the powerful global modelling capability of the Swin Transformer, we propose a new RS image semantic segmentation network. Swin Transformer is used as an encoder to extract global features. To prevent the loss of local details in the encoding stage, one LKA-PM (LKA-Patch Merging) module is designed to extract local detail information by decomposing large kernel convolutions, thereby enhancing the feature representation ability of small objects. Furthermore, to decode global and local features and obtain multi-scale information, one WORC (Weighted Operation Re-parameterization Convolution) module is designed to improve the adaptability and accuracy of the model to different RS images. Finally, to preserve spatial details during decoding and reduce the problem of boundary blurring caused by occlusion in RS imagery, the Coordswin module is designed to provide coordinate information and improve the discrimination ability of similar objects. Experimental results demonstrate that our CW-SwinUNet achieves excellent segmentation performance (91.8 % F1 and 85.2 % MIoU) on the Vaihingen dataset and (93.8 % F1 and 87.9 % MIoU) on Potsdam datasets, surpassing other state-of-the-art methods. The source code of the proposed CW-SwinUNet is available at https://github.com/xmy1135/cwswin/tree/main.

The presence of shadows in high-resolution (HR) remote-sensing images reduces object detection accuracy. To address this problem, in this paper, we proposed a deep neural network algorithm for shadow detection by using the AISD and SSAD remote-sensing shadow image datasets. To improve the ability to extract spatial information from feature maps, we developed a cross-spatial attention module that focuses on semantic information in the horizontal and vertical directions at each position point on the remote-sensing image. This module overcomes the limitations of existing technologies in accurately judging small areas and suspected shadow areas and in missing or incorrectly detected shadow areas. In addition, to improve the ability to extract shadow features and the accuracy of shadow detection in remote-sensing images, we developed a channel attention module that assigns more attention to channels that conform to the shadow color characteristics. The network architecture comprises an encoder – decoder structure, with ResNeXt50 used as the backbone for the encoder and a multi resolution parallel fusion (MRPF) designed for the decoder; cross-spatial and channel attention were incorporated into the decoder unit. Experimental results demonstrated the superior performance of the proposed algorithm, with an F1 score of 92.6% for the shadow category on the test set, thus, outperforming other algorithms and making the proposed method an effective solution for shadow detection in HR remote-sensing images.