Real‐time Deep Radiance Reconstruction from Imperfect Caches

Global illumination dramatically improves realistic appearance of rendered scenes, but usually it is neglected in VR systems due to its high costs. In this work we present an efficient global illumination solution specifically tailored for those CAVE applications, which require an immediate response for dynamic light changes and allow for free motion of the observer, but involve scenes with static geometry. As an application example we choose the car interior modeling under free driving conditions. We illuminate the car using dynamically changing High Dynamic Range (HDR) environment maps and use the Precomputed Radiance Transfer (PRT) method for the global illumination computation. We leverage the PRT method to handle scenes with non-trivial topology represented by complex meshes. Also, we propose a hybrid of PRT and final gathering approach for high-quality rendering of objects with complex Bi-directional Reflectance Distribution Function (BRDF). We use this method for predictive rendering of the navigation LCD panel based on its measured BRDF. Since the global illumination computation leads to HDR images we propose a tone mapping algorithm tailored specifically for the CAVE. We employ head tracking to identify the observed screen region and derive for it proper luminance adaptation conditions, which are then used for tone mapping on all walls in the CAVE. We distribute our global illumination and tone mapping computation on all CPUs and CPUs available in the CAVE, which enables us to achieve interactive performance even for the costly final gathering approach.

Precomputed radiance transfer (PRT) methods established for handling global illumination (GI) of objects from area lights in real time and many techniques proposed for rotating the light using linear algebra rotation matrices. Rotating area lights efficiently are crucial part for computer graphics since it is one of the main components of real‐time rendering. Matrices commonly used for handling such rotations are not quite efficient and require high memory consumption; as a result, the need for proposing new more efficient rotation algorithms has been established. In this work, we use the conformal geometric algebra (CGA) as the mathematical background for “GI in real‐time” under distant Image‐based lighting (IBL) illumination, for diffuse surfaces with self‐shadowing by efficiently rotating the environment light using CGA entities. Our work is based on spherical harmonics (SH), which are used for approximating natural, area‐light illumination as irradiance maps. Our main novelty is that we extend the PRT algorithm by representing SH for the first time with CGA.The main intuition is that SH of band index 1 are represented using CGA entities and SH with band index larger than 1 are represented in terms of CGA‐SH of band 1. Specifically, we propose a new method for representing SH with CGA entities and rotating SH by rotating CGA entities. In this way, we can visualize the SH rotations, rotate them faster than rotation matrices, and we provide a unique visual representation and intuition regarding their rotation, in stark contrast to usual rotation matrices, and we achieve consistently better visual results from Ivanic rotation matrices during light rotation. Via our CGA expressed SH, we provide a significant boost on the PRT algorithm since we represent SH rotations by CGA rotors (4 numbers) as opposed to 9 × 9 sparse matrices that are usually used. With our algorithm, we pave the way for including scaling (dilation) and translation of light coefficients using CGA motors.

Interactive rendering of realistic objects under general lighting models poses three principal challenges. Handling complex light transport phenomena like shadows, inter-reflections, caustics and sub-surface scattering is difficult to do in real time. Integrating these effects over large area light sources compounds the difficulty, and finally real objects have complex spatially-varying BRDF's. Precomputed Radiance Transfer (PRT) encapsulates a family of techniques that partially addresses these challenges. PRT is an active of area of research that has relevance to both the academic research community and practitioners of interactive computer graphics. This technique and its variants are being actively investigated in the game development community and there is quite a lot of interest due to the recent appearance of PRT techniques in games such as Halo 2.This course covers these techniques, compares them and discusses their various strengths and weaknesses. A more rigorous derivation directly from the rendering equation is presented along with practical implementation details, both of which are generally not included in technical papers. After introducing the necessary foundation (rendering equation, basis functions, etc.), we begin with simple PRT for diffuse objects. We continue with general PRT using the concept of transfer matrices, which allow for arbitrary reflectance models. The possible choices for basis functions are discussed as well. Different light source representations are presented and compared. Finally, we discuss practical issues with PRT, such as data compression, spatial sampling, normal mapping, precomputation, and more. By the end of the course the audience will be able to pick the right algorithm for their needs and will hopefully have gained some of the unpublished insights the speakers have gained by working in this area.

3D immersive scene generation is a challenging yet critical task in computer vision and graphics. A desired virtual 3D scene should 1) exhibit omnidirectional view consistency, and 2) allow for large-range exploration in complex scene hierarchies. Existing methods either rely on successive scene expansion via inpainting or employ panorama representation to represent large FOV scene environments. However, the generated scene suffers from semantic drift during expansion and is unable to handle occlusion among scene hierarchies. To tackle these challenges, we introduce LayerPano3D, a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt. Our key insight is to decompose a reference 2D panorama into multiple layers at different depth levels, where each layer reveals the unseen space from the reference views via diffusion prior. LayerPano3D comprises multiple dedicated designs: 1) We introduce a new panorama dataset Upright360 , comprising 9k high-quality and upright panorama images, and finetune the advanced Flux model on Upright360 for high-quality, upright and consistent panorama generation related tasks. 2) We pioneer the Layered 3D Panorama as underlying representation to manage complex scene hierarchies and lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes with unconstrained viewing paths. Extensive experiments demonstrate that our framework generates state-of-the-art 3D panoramic scene in both full view consistency and immersive exploratory experience. We believe that LayerPano3D holds promise for advancing 3D panoramic scene creation with numerous applications. More examples please visit our webpage: ys-imtech.github.io/projects/LayerPano3D/.

Real time global illumination (GI) is a difficult and steadily researched area. Advances in the field could potentially benefit virtual reality applications by increasing users' sense of presence. In immersive virtual environments (IVE) like CAVEs, applications must support perspective-corrected stereoscopic rendering. Dmitriev et al. [2004] performed GI in a CAVE using Precomputed Radiance Transfer, which requires a static scene. Mortensen et al. [2007] also performed GI in a CAVE using Virtual Light Fields which did not allow moving lights or geometry. We present our attempt to find GI techniques that support dynamic lights and scene geometry in our 6-sided CAVE-like IVE (DRIVE6). We implemented two separate illumination techniques: GPU photon mapping (GPM) and multiresolution splatting for indirect illumination (MSII). Each technique makes trade-offs between image quality and speed, and appropriate use of each depends on the needs of the application. Anecdotal evidence suggests that these techniques increase the sense of presence, warranting formal study. A user study is planned.

We present a lightweight framework composed of two extensions to introduce precomputed radiance transfer (PRT) to X3D. Depending on the basis function, PRT is used to simulate shading, complex light transfer and effects like subsurface scattering and caustics in real time by projecting incident light and transfer functions into frequency space, which allows the solution of the integration of the rendering equation through a simple dot product. Light transfer from incident to transferred radiance thus becomes a matter of solving dot products of coefficient vectors, which can be easily evaluated on the GPU. In this paper, we identify the minimal set of components needed for an X3D application to use spherical harmonic based PRT, and discuss how further enhancements to PRT can be implemented, like new basis functions and other features.

City models visualisation, buildings, structures and volumetric information, is an important task in Computer Graphics and Urban Planning. The different formats and data sources involved in the visualisation make the development of applications a big challenge. We present a homogeneous web visualisation framework using X3DOM and MEDX3DOM for the visualisation of these urban objects. We present an integration of different declarative data sources, enabling the utilization of advanced visualisation algorithms to render the models. It has been tested with a city model composed of buildings from the Madrid University Campus, some volumetric datasets coming from Air Quality Models and 2D layers wind datasets. Results show that the visualisation of all the urban models can be performed in real time on the Web. An HTML5 web interface is presented to the users, enabling real time modifications of visualisation parameters.

Creating realistic animations of human faces is still a challenging task in computer graphics. While computer graphics (CG) models capture much variability in a small parameter vector, they usually do not meet the necessary visual quality. This is due to the fact, that geometry‐based animation often does not allow fine‐grained deformations and fails in difficult areas (mouth, eyes) to produce realistic renderings. Image‐based animation techniques avoid these problems by using dynamic textures that capture details and small movements that are not explained by geometry. This comes at the cost of high‐memory requirements and limited flexibility in terms of animation because dynamic texture sequences need to be concatenated seamlessly, which is not always possible and prone to visual artefacts. In this study, the authors present a new hybrid animation framework that exploits recent advances in deep learning to provide an interactive animation engine that can be used via a simple and intuitive visualisation for facial expression editing. The authors describe an automatic pipeline to generate training sequences that consist of dynamic textures plus sequences of consistent three‐dimensional face models. Based on this data, they train a variational autoencoder to learn a low‐dimensional latent space of facial expressions that is used for interactive facial animation.

Normal mapping is a variant of bump mapping that is commonly used in computer games. It models complex surface variations by explicitly storing a surface normal in a texture map. However, it has not been used with precomputed radiance transfer (PRT), a technique for modeling an object's response to a parameterized model of lighting, which enables interactive rendering of complex global illumination effects such as soft shadows and interreflections. This paper presents several techniques that effectively combine normal mapping and precomputed radiance transfer for rigid objects. In particular, it investigates representing the convolved radiance function in various bases and borrowing concepts from the separable decomposition of BRDF's.

The creation and rendering of realistic water scenes is one of the challenging tasks in Computer Graphics. To reproduce the illumination and colour inside water bodies an algorithm capable of dealing with media with anisotropic and multiple scattering has to be used. We have developed a simulation system based in the discrete ordinates method to solve the problem of light transport in general participating media. We discuss its application to the rendering of images of natural waters, a difficult task due to the different components that determine their optical behaviour. A couple of simple images calculated in different waters are presented. Results indicate the relevant role played by the spectral behaviour of the absorption and scattering coefficients and by the correct treatment of the phase function in the process of image generation.

Finding distortion-minimizing homeomorphisms between surfaces of arbitrary genus is a fundamental task in computer graphics and geometry processing. We propose a simple method utilizing intrinsic triangulations, operating directly on the original surfaces without going through any intermediate domains such as a plane or a sphere. Given two models A and B as triangle meshes, our algorithm constructs a Compatible Intrinsic Triangulation (CIT), a pair of intrinsic triangulations over A and B with full correspondences in their vertices, edges and faces. Such a tessellation allows us to establish consistent images of edges and faces of A's input mesh over B (and vice versa) by tracing piecewise-geodesic paths over A and B. Our algorithm for constructing CITs, primarily consisting of carefully designed edge flipping schemes, is empirical in nature without any guarantee of success, but turns out to be robust enough to be used within a similar second-order optimization framework as was used previously in the literature. The utility of our method is demonstrated through comparisons and evaluation on a standard benchmark dataset.

The realistic rendering of fire scene is one of the most challenging tasks in computer graphics. Based on GPU, the rendering of fire and smoke is studied by using particle system. The fire is treated as three parts: fireball, flame and ember, and every part is simulated by using the respective particle system. The light reflection and scattering are realized to improve the realism of smoke simulation. The method developed in the paper has been applied successfully in the scene of maritime search and rescue simulator.

Fuzzy geometric object modelling has been one of the fundamental tasks in computer graphics. In this paper, we investigated and developed techniques for modelling fuzzy objects based on fuzzy set theory. With the proposed techniques any conventional geometric objects, such as points, lines, triangles, and curved surfaces can all be easily extended as certain kind of fuzzy sets to represent geometric objects with fuzzy boundaries. In addition, experimental results are presented to demonstrate how 3D fuzzy shapes represented in this way can be rendered efficiently and effectively in multiple passes using shader programs running entirely on programmable GPUs.

Point cloud registration is a fundamental task in computer graphics, and more specifically, in rigid and non-rigid shape matching. The rigid shape matching problem can be formulated as the problem of simultaneously aligning and labelling two point clouds in 3D so that they are as similar as possible. We name this problem the Procrustes matching (PM) problem. The non-rigid shape matching problem can be formulated as a higher dimensional PM problem using the functional maps method. High dimensional PM problems are difficult non-convex problems which currently can only be solved locally using iterative closest point (ICP) algorithms or similar methods. Good initialization is crucial for obtaining a good solution. We introduce a novel and efficient convex SDP (semidefinite programming) relaxation for the PM problem. The algorithm is guaranteed to return a correct global solution of the problem when matching two isometric shapes which are either asymmetric or bilaterally symmetric. We show our algorithm gives state of the art results on popular shape matching datasets. We also show that our algorithm gives state of the art results for anatomical classification of shapes. Finally we demonstrate the power of our method in aligning shape collections.

3D Shape Deformation Using Stick Figures