Lagrangian Hashing for Compressed Neural Field Representations

Neural fields are a new (and old!) approach to solving problems over spacetime via first-order optimization of a neural network. Over the past three years, combining neural fields with classic computer graphics approaches have allowed us to make significant advances in solving computer vision problems like scene reconstruction. I will present recent work that can reconstruct indoor scenes for photorealistic interactive exploration using new scalable hybrid neural field representations. This has applications where any real-world place needs to be digitized, especially for visualization purposes.

Neural radiance fields and neural reflectance fields are novel deep learning methods for generating novel views of 3D scenes from 2D images. To extend the neural scene representation techniques to complex underwater environments, beyond neural reflectance fields underwater (BNU) was proposed, which considers the relighting conditions of on-aboard light sources by using neural reflectance fields, and approximates the attenuation and backscatter effects of water with an additional constant. Because the quality of the neural representation of underwater scenes is critical to downstream tasks such as marine surveying and mapping, the model reliability should be considered and evaluated. However, current neural reflectance models lack the ability of quantifying the uncertainty of underwater scenes that are not directly observed during training, which hinders their widespread use in the field of underwater unmanned autonomous navigation. To address this issue, we introduce an ensemble strategy to BNU that quantifies cognitive uncertainty in color space and unobserved regions with the expectation and variance of RGB values and termination probabilities along the ray. We also employ a regularization method to smooth the density of the underwater neural reflectance model. The effectiveness of the present method is demonstrated in numerical experiments.

Neural representations (NRs), such as neural fields and 3D Gaussians, effectively model volumetric data in computed tomography (CT) but suffer from severe artifacts under sparse-view settings. To address this, we propose DiffNR, a novel framework that enhances NR optimization with diffusion priors. At its core is SliceFixer, a single-step diffusion model designed to correct artifacts in degraded slices. We integrate specialized conditioning layers into the network and develop tailored data curation strategies to support model finetuning. During reconstruction, SliceFixer periodically generates pseudo-reference volumes, providing auxiliary 3D perceptual supervision to fix underconstrained regions. Compared to prior methods that embed CT solvers into time-consuming iterative denoising, our repair-and-augment strategy avoids frequent diffusion model queries, leading to better runtime performance. Extensive experiments show that DiffNR improves PSNR by 3.99 dB on average, generalizes well across domains, and maintains efficient optimization.

Efficient reasoning about the semantic, spatial, and temporal structure of a scene is a crucial prerequisite for autonomous driving. We present NEural ATtention fields (NEAT), a novel representation that enables such reasoning for end-to-end imitation learning models. NEAT is a continuous function which maps locations in Bird’s Eye View (BEV) scene coordinates to waypoints and semantics, using intermediate attention maps to iteratively compress high-dimensional 2D image features into a compact representation. This allows our model to selectively attend to relevant regions in the input while ignoring information irrelevant to the driving task, effectively associating the images with the BEV representation. In a new evaluation setting involving adverse environmental conditions and challenging scenarios, NEAT outperforms several strong baselines and achieves driving scores on par with the privileged CARLA expert used to generate its training data. Furthermore, visualizing the attention maps for models with NEAT intermediate representations provides improved interpretability.

This paper proposes an IP forwarding information base (FIB) encoding leveraging an emerging data structure called a learned index , which uses machine learning to associate key-position pairs in a key-value store. A learned index for FIB lookups is expected to yield a more compact representation and faster lookups compared to existing FIBs based on tries or hash tables, at the cost of efficient FIB updates, which is difficult to support with a learned index. We optimize our implementation for lookup speed, exploiting that for efficient FIB lookups it is enough to approximate the key-position pairs with a piece-wise linear function, instead of having to learn the key-position pairs. The experiments using real BGP routing information snapshots suggest that the size of the proposed FIB is compact and lookup speed is sufficiently fast regardless of the length of matched prefixes.

This paper proposes a strategy for visual perception in the context of autonomous driving. Humans, when not distracted or drunk, are still the best drivers you can currently find. For this reason, we take inspiration from two theoretical ideas about the human mind and its neural organization. The first idea concerns how the brain uses structures of neuron ensembles that expand and compress information to extract abstract concepts from visual experience and code them into compact representations. The second idea suggests that these neural perceptual representations are not neutral but functional to predicting the future state of affairs in the environment. Similarly, the prediction mechanism is not neutral but oriented to the planning of future action. We identify within the deep learning framework two artificial counterparts of the aforementioned neurocognitive theories. We find a correspondence between the first theoretical idea and the architecture of convolutional autoencoders, while we translate the second theory into a training procedure that learns compact representations which are not neutral but oriented to driving tasks, from two distinct perspectives. From a static perspective, we force separate groups of neural units in the compact representations to represent specific concepts crucial to the driving task distinctly. From a dynamic perspective, we bias the compact representations to predict how the current road scenario will change in the future. We successfully learn compact representations that use as few as 16 neural units for each of the two basic driving concepts we consider: cars and lanes . We maintain the two concepts separated in the latent space to facilitate the interpretation and manipulation of the perceptual representations. The source code for this paper is available at https://github.com/3lis/rnn_vae .

Unsupervised reconstruction of accelerated cardiac cine MRI using neural fields.

Sparse-view computed tomography (CT) has been attracting attention for its reduced radiation dose and scanning time. However, analytical image reconstruction methods suffer from streak artifacts due to insufficient projection views. Recently, various deep learning-based methods have been developed to solve this ill-posed inverse problem. Despite their promising results, they are easily overfitted to the training data, showing limited generalizability to unseen systems and patients. In this work, we propose a novel streak artifact reduction algorithm that provides a system- and patient-specificsolution. Motivated by the fact that streak artifacts are deterministic errors, we regenerate the same artifacts from a prior CT image under the same system geometry. This prior image need not be perfect but should contain patient-specific information and be consistent with full-view projection data for accurate regeneration of the artifacts. To this end, we use a coordinate-based neural representation that often causes image blur but can greatly suppress the streak artifacts while having multiview consistency. By employing techniques in neural radiance fields originally proposed for scene representations, the neural representation is optimized to the measured sparse-view projection data via self-supervised learning. Then, we subtract the regenerated artifacts from the analytically reconstructed original image to obtain the final correctedimage. To validate the proposed method, we used simulated data of extended cardiac-torso phantoms and the 2016 NIH-AAPM-Mayo Clinic Low-Dose CT Grand Challenge and experimental data of physical pediatric and head phantoms. The performance of the proposed method was compared with a total variation-based iterative reconstruction method, naive application of the neural representation, and a convolutional neural network-based method. In visual inspection, it was observed that the small anatomical features were best preserved by the proposed method. The proposed method also achieved the best scores in the visual information fidelity, modulation transfer function, and lung nodulesegmentation. The results on both simulated and experimental data suggest that the proposed method can effectively reduce the streak artifacts while preserving small anatomical structures that are easily blurred or replaced with misleading features by the existing methods. Since the proposed method does not require any additional training datasets, it would be useful in clinical practice where the large datasets cannot becollected.

Sparse-view CBCT reconstruction using meta-learned neural attenuation field and hash-encoding regularization.

Neural radiance field (NeRF) is an emerging computer graphics task that is used for 3-D modeling and rendering in the metaverse, providing a user-friendly and immersive experience. It successfully replaces conventional 3-D modeling methods such as photogrammetry thanks to its simplicity, photo-realistic rendering quality, and versatility. However, it has limitations to be accelerated on mobile augmented reality (AR)/virtual reality (VR) devices due to its memory-intensive hash encoding and extensive computational load. Previous NeRF application specified integrated circuits (ASICs) only supported high frame-rate rendering while being incapable of immediate 3-D model creation. In this article, we present neural graphics processing unit (NeuGPU) to achieve both instant 3-D modeling and real-time rendering using NeRF with three key features. First, segmented hashing with spatial pruning (SHSP) partitions hash table (HT) into multiple segments and reduces 66% of external memory access (EMA). Spatial management unit (SMU) supports SHSP operation through three types of sub-block management. Second, attention-based hybrid interpolation unit (AHIU) optimizes heterogeneous memory access characteristics of hash-based NeRF, achieving 56.4% power reduction. Third, similarity-sparsity skipping (S3) core supports energy-efficient multi-layer perceptron (MLP) operation by applying coarse-grained skip for both similar and sparse data. As a result, NeuGPU is fabricated in a 28-nm process and occupies a 20.25-mm2 die area. Evaluated on the synthetic NeRF dataset, NeuGPU finally achieves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8.7{\times }$ </tex-math></inline-formula> faster modeling and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$231.4{\times }$ </tex-math></inline-formula> smaller energy per iteration compared with edge GPU even though it has <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$31{\times }$ </tex-math></inline-formula> smaller external memory bandwidth.

The discrete Fourier transform (DFT) can be considered as an observing system, which has an input f, an output F, and a response with additive noise E. In many applications, part of the frequency spectrum/frequency information is missing or unavailable due to the passage of the time-limited signal through a band-limited system, for example, the discrete Fourier system. We suggest improving the resolution of the reconstruction of signals and images using a novel approach for the solution of the discrete Fourier system and by image enhancement. We note that the reconstruction of a time-limited signal can be simply realized by only using either the real part or the imaginary part of the DFT matrix. Therefore, based on the study of the special structure of the real and imaginary parts of the discrete Fourier matrix, a fast direct computational method is developed that utilizes explicit formulas for the truncated singular value decomposition (TSVD) obtained recently by the authors. For improving the resolution of the reconstructions, enhancement by logarithm transform is applied. This fast direct computational method is superior to other direct methods such as LU decomposition, QR decomposition, classical SVD and classical TSVD. The explicit TSVD along with the enhancement can be considered as a useful tool for signal and image reconstructions. Numerical tests for signal and image reconstructions and enhancements are given as well.

LiDAR Mapping has been a long-standing problem in robotics. Recent progress in neural implicit representation has brought new opportunities to robotic mapping. In this letter, we propose the multi-volume neural feature fields, called NF-Atlas, which bridge the neural feature volumes with pose graph optimization. By regarding the neural feature volume as pose graph nodes and the relative pose between volumes as pose graph edges, the entire neural feature field becomes both <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">locally rigid</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">globally elastic</i> . Locally, the neural feature volume employs a sparse feature Octree and a small MLP to encode the signed distance function (SDF) of the submap with an option of semantics. Learning the map using this structure allows for end-to-end solving of maximum a posteriori (MAP) based probabilistic mapping. Globally, the map is built volume by volume independently, avoiding catastrophic forgetting when mapping incrementally. Furthermore, when a loop closure occurs, with the elastic pose graph based representation, only updating the origin of neural volumes is required without remapping. Finally, these functionalities of NF-Atlas are validated. Thanks to the sparsity and the optimization based formulation, NF-Atlas shows competitive performance in terms of accuracy, efficiency and memory usage on both simulation and real-world datasets.

High-resolution imagery is limited by sensor technology, atmospheric effects, and acquisition costs. This is a well-known challenge in satellite remote sensing, but it also applies to ground-level imaging with handheld devices such as smartphones. Super-resolution seeks to overcome these limitations by enhancing image resolution algorithmically. Single-image super-resolution, however, is an ill-posed inverse problem and therefore depends on strong priors, typically learned from high-resolution training data or imposed through auxiliary information such as high-resolution guidance from another modality. While these methods often produce visually appealing results, they are prone to hallucinating structures that do not reflect the true scene content.Multi-image super-resolution (MISR) addresses this issue by exploiting multiple low-resolution views of the same scene that are captured with sub-pixel shifts. In this work, we introduce SuperF, a test-time optimization approach for MISR based on coordinate-based neural networks, also known as neural fields. By representing images as continuous signals using implicit neural representations (INRs), neural fields are well suited for reconstructing high-resolution images from multiple aligned observations. The central idea of SuperF is to share a single INR across all low-resolution frames while jointly optimizing the image representation and the sub-pixel alignment between frames.Compared to prior INR-based approaches adapted from burst fusion and layer separation, SuperF directly parameterizes the sub-pixel alignment using optimizable affine transformation parameters and performs the optimization on a super-sampled coordinate grid corresponding to the target output resolution. We evaluate the proposed method on simulated bursts of satellite imagery as well as on ground-level images captured with handheld cameras, and observe consistent improvements for upsampling factors of up to 8. A key advantage of SuperF is that it operates entirely at test time and does not rely on any high-resolution training data.

Existing Simultaneous Localization and Mapping (SLAM) approaches are limited in their scalability due to growing map size in long-term robot operation. Moreover, processing such maps for localization and planning tasks leads to the increased computational resources required onboard. To address the problem of memory consumption in long-term operation, we develop a novel real-time SLAM algorithm, MeSLAM, that is based on neural field implicit map representation. It combines the proposed global mapping strategy, including neural networks distribution and region tracking, with an external odometry system. As a result, the algorithm is able to efficiently train multiple networks representing different map regions and track poses accurately in large-scale environments. Experimental results show that the accuracy of the proposed approach is comparable to the state-of-the-art methods (on average, 6.6 cm on TUM RGB-D sequences) and outperforms the baseline, iMAP*. Moreover, the proposed SLAM approach provides the most compact-sized maps without details distortion (1.9 MB to store 57 m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ) among the state-of-the-art SLAM approaches.

Differentiable rendering techniques have received significant attention from both industry and academia for novel view synthesis or for reconstructing shapes and materials from one or multiple input photographs. These techniques are used to propagate gradients from image pixel colors back to scene parameters. The obtained gradients can then be used in various optimization algorithms to reconstruct the scene representation or can be further propagated into a neural network to learn the scene’s neural representations. In this work, we provide a brief taxonomy of existing popular differentiable rendering methods, categorizing them based on the primary rendering algorithms employed: physics-based differentiable rendering (PBDR), methods based on neural radiance fields (NeRFs), and methods based on 3D Gaussian splatting (3DGS). Since there are already several reviews for NeRF-based or 3DGS-based differentiable rendering methods but almost zero for physics-based differentiable rendering, we place our main focus on PBDR and, for completeness, only review several improvements made for NeRF and 3DGS in this survey. Specifically, we provide introductions to the theories behind all three categories of methods, a benchmark comparison of the performance of influential works across different aspects, and a summary of the current state and open research problems. With this survey, we seek to welcome new researchers to the field of differentiable rendering, offer a useful reference for key influential works, and inspire future research through our concluding section.