LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation

The evolution of generative models has been quite rapid, and this has greatly impacted the field of 3D scene generation. It was previously impossible to build such detailed 3D scenes automatically and with relative simplicity because the process demanded considerable human input alongside significant computational power. At the same time, it could be a labor-intensive task, especially when a large number of scenes needed to be generated. New generative synthesis methods such as diffusion models and GANs have made it much easier and more efficient to synthesize high-quality 3D scenes. Of these, the diffusion models have turned out to excel at crafting diverse and highly complex multi-object 3D scenes that are photorealistic and provide a new record in scene generation. Such technologies have enabled new opportunities across industries such as VR, augmented reality, gaming, robotics, and automation, where approximate and more realistic 3D models are increasingly becoming useful. The diffusion models, notably, show comparatively high performance in creating complex spatial patterns and a high level of detail, which is beneficial for realistic applications. This paper discusses the basics, uses, limitations, and innovations of current generative models, with an emphasis on diffusion models, with a view to applying them mainly to 3D scene generation. Thus, this review highlights the possibilities and oversights of diffusion models to demonstrate the possibilities of the future of automated 3D content creation across several technological domains.

We present a generative neural network that enables us to generate plausible 3D indoor scenes in large quantities and varieties, easily and highly efficiently. Our key observation is that indoor scene structures are inherently hierarchical . Hence, our network is not convolutional; it is a recursive neural network, or RvNN. Using a dataset of annotated scene hierarchies, we train a variational recursive autoencoder , or RvNN-VAE, which performs scene object grouping during its encoding phase and scene generation during decoding. Specifically, a set of encoders are recursively applied to group 3D objects based on support, surround, and co-occurrence relations in a scene, encoding information about objects’ spatial properties, semantics , and relative positioning with respect to other objects in the hierarchy. By training a variational autoencoder (VAE), the resulting fixed-length codes roughly follow a Gaussian distribution. A novel 3D scene can be generated hierarchically by the decoder from a randomly sampled code from the learned distribution. We coin our method GRAINS, for Generative Recursive Autoencoders for INdoor Scenes. We demonstrate the capability of GRAINS to generate plausible and diverse 3D indoor scenes and compare with existing methods for 3D scene synthesis. We show applications of GRAINS including 3D scene modeling from 2D layouts, scene editing, and semantic scene segmentation via PointNet whose performance is boosted by the large quantity and variety of 3D scenes generated by our method.

Abstract3D scene understanding and generation are to reconstruct the layout of the scene and each object from an RGB image, estimate its semantic type in 3D space and generate a 3D scene. At present, the 3D scene generation algorithm based on deep learning mainly recovers the 3D scene from a single image. Due to the complexity of the real environment, the information provided by a single image is limited, and there are problems such as the lack of single-view information and the occlusion of objects in the scene. In response to the above problems, we propose a 3D scene generation framework SGMT, which realizes multi-view position information fusion and reconstructs the 3D scene from multi-view video time series data to compensate for the missing object position in existing methods. We demonstrated the effectiveness of multi-view scene generation of SGMT on the UrbanScene3D and SUNRGBD dataset and studied the influence of SGCN and joint fine-tuning. In addition, we further explored the transfer ability of the SGMT between datasets and discussed future improvements.Keywords3D scene generationMulti-view fusionMulti-view time series data

Computer-assisted surveillance of complex environments is becoming more and more interesting, thanks to significant improvements in real-time signal processing. In the surveillance research field applied to public areas, crowd monitoring is very useful but presents particularly complex problems. Recognizing human bodies and tracking their movements in complex real scenes by using a sequence of images are among the most difficult tasks in computer vision. The main difficulties arise both from the high level of shape variability associated with moving human bodies, and from the complexity of environmental real scenes. A method for tracking human motion in 3D real scenes by means of Kalman filtering is presented It is based on the introduction of a suitable mathematical model for a human body moving in a 3D scene and interacting with the surrounding environment. The developed refined static and dynamic models allow the system to be accurate and robust. Due to its real-time functioning, accuracy and robustness, the method can be used in surveillance systems devoted to assure public safety.

Computer Vision, encompassing 3D Vision and 3D scene Reconstruction, is a field of importance to real-world problems involving 3D views of scenes. The goal of the proposed system is to retrieve 3D scenes from the database, and further augment the scenes in an iterative manner based on the user’s commands to finally produce the required output 3D scenes, in the form of a suggestive interface. The process is done recursively to facilitate additions and deletions, until the desired scene is generated. In addition to synthesizing the required 3D indoor scenes from text, a speech recognition system has been integrated with the system that will enable the users to choose from either modes of input. The application includes the projection and rendering of 3D scenes which will enable a 360-\({^\circ }\) view of the scene. The robustness of scene generation and quick retrieval of scenes will promote the usage of this work in avenues such as story telling, interior designing, and as a helpful educational tool for autistic children.

The goal of this thesis is to present my research contributions towards handling partially visible input in different visual tasks, comprising 360° RGB-D panorama image completion, 3D scene decomposition, and 3D amodal reconstruction. This thesis consists of three main pieces of work, each of which presents a novel learning-based approach for recovering complete visual representations from partially observed data. From different perspectives, each work demonstrates the novelty and superiority of the proposed method on utilizing occluded, inconsistent, and partially visible input for 3D generation. Chapter 2 provides a comprehensive literature review of 3D visual synthesis and generation, establishing the theoretical foundation for the subsequent contributions. This chapter systematically examines the evolution of 3D visual representations, followed by exploring three fundamental tasks in 3D computer vision: novel-view synthesis for generating photorealistic images from unseen viewpoints, 3D object generation, and 3D scene decomposition & generation. By analyzing the strengths and limitations of existing methodologies, we identify key research gaps and challenges that motivate the novel approaches presented in the following chapters. Chapter 3 presents a method for completing 360° panoramic scenes from limited narrow field-of-view (NFoV) input. We introduce a learning-based framework that treats panorama outpainting as a dual-modal latent diffusion process. During training, the model jointly encodes full resolution RGB and depth panoramas into a unified latent space and learns to denoise corrupted latents back into coherent RGB-D outputs. Two novel strategies - horizontal cyclic consistency through simulated camera rotations and explicit alignment of panorama boundaries during sampling - ensure that the generated panorama wraps seamlessly from left to right. At test time, given only a NFoV RGB crop, the system hallucinates both the missing color regions and the associated depth values, then decodes them into a full-resolution RGB-D panorama. Extensive experiments on public indoor datasets demonstrate that this approach produces semantically consistent, geometrically accurate 360° outputs, outperforming prior GAN-based and autoregressive diffusion baselines in both RGB fidelity and depth prediction. Chapter 4 focuses on decomposing complex indoor scenes into individual 3D object surfaces using only noisy multi-view 2D segmentation masks, without relying on any ground-truth 3D annotations. We present a neural implicit representation in which the network outputs multiple signed distance function (SDF) channels, each interpreted as the occupancy probability of a distinct object. A novel clustering-oriented loss combines three components: (1) a differentiation term that pushes different output channels apart, (2) a one-hot constraint that encourages each surface point to activate exactly a single channel, and (3) a regularization term to prevent trivial collapse. During training, rays sampled from multiple RGB-D viewpoints are rendered through the implicit network and matched against the corresponding 2D masks; the clustering loss then guides voxels from the same object to share one channel, while voxels from different objects occupy different channels. Experimental results on multiple benchmarks show that this weakly supervised pipeline yields high-quality per-object 3D decompositions, matching or exceeding the performance of fully supervised alternatives. Chapter 5 addresses 3D object reconstruction under occlusion. We describe a framework that integrates occlusion-aware attention mechanisms into a pretrained 3D generative diffusion backbone. Given one or more RGB views and their automatically extracted visibility and occlusion masks, the model first uses mask-weighted cross-attention to focus on observed regions during latent denoising, then employs a specialized occlusion attention layer to hallucinate the geometry of hidden surfaces. By performing diffusion directly in a 3D Gaussian implicit space, the system simultaneously recovers the geometry of visible parts and infers missing occluded regions. When multiple views are available, an adaptive fusion strategy orders features by apparent visibility to further enhance reconstruction fidelity. Comprehensive evaluations on multiple 3D datasets and real-scene occlusion benchmarks demonstrate that this end-to-end approach consistently outperforms two-stage baselines which first perform 2D amodal completion and then 3D reconstruction. Together, these three contributions form a cohesive research pipeline for handling partially visible inputs of different types: beginning with completing missing panorama content from NFoV images, moving to weakly supervised object-level decomposition of 3D scenes, and culminating in occlusion-aware single-stage 3D object generation. By integrating latent diffusion, neural implicit representations, and occlusion-aware attention, this thesis demonstrates how to robustly recover complete visual representations from limited or inconsistent observations, thereby advancing the state-of-the-art in real-world 3D vision tasks.

3D scene generation for zero-shot learning using ChatGPT guided language prompts

Generating 3D scene and animation plays an important role in many domain of our daily life. However, the difficulty of its creating manually and its consumption of effort and time which pushed many and many researchers to shift their interest into the field of automatic 3D scene generation and precisely from natural language description. The results obtained during the last few years revived hopes for the widespread availability of 3D scenes in our daily activities. In this survey, we will give an overview of the most popular “Text” to “3D scenes” converting systems and their architectures highlighting their strengths and weaknesses.

Monocular depth estimation is a fundamental yet challenging task in computer vision as depth information will be lost when 3D scenes are mapped to 2D images. Although deep learning-based methods have led to considerable improvements for this task in a single image, most existing approaches still fail to overcome this limitation. Supervised learning methods model depth estimation as a regression problem and, as a result, require large amounts of ground truth depth data for training in actual scenarios. Unsupervised learning methods treat depth estimation as the synthesis of a new disparity map, which means that rectified stereo image pairs need to be used as the training dataset. Aiming to solve such problem, we present an encoder-decoder based framework, which infers depth maps from monocular video snippets in an unsupervised manner. First, we design an unsupervised learning scheme for the monocular depth estimation task based on the basic principles of structure from motion (SfM) and it only uses adjacent video clips rather than paired training data as supervision. Second, our method predicts two confidence masks to improve the robustness of the depth estimation model to avoid the occlusion problem. Finally, we leverage the largest scale and minimum depth loss instead of the multiscale and average loss to improve the accuracy of depth estimation. The experimental results on the benchmark KITTI dataset for depth estimation show that our method outperforms competing unsupervised methods.

The purpose of this study is to explore the application of 3D Gaussian Splatting technology in automated driving training. In reality, the training of automatic driving models will produce great labor and economic consumption, but the use of virtual 3D scenes can significantly reduce this consumption, and at the same time have better training effects. Therefore, how to better use 3D for the training of automatic driving models will also be the future development direction of the field of automatic driving. In this paper, some major 3D Gaussian based dynamic scene modeling methods are summarized and some improvement schemes are proposed, including scene segmentation, new scene generation and dynamic scene rendering. This paper hopes to summarize and improve the current research, and finally propose that the generation target scene can be customized according to the training requirements of users' automatic driving models, so as to achieve more efficient and practical 3D scene generation.

Three-dimensional (3D) scene models are essential for how humans and intelligent systems perceive, reason about, and interact with the world. From digital avatars and product design to indoor environments and urban landscapes, 3D data provides semantic and spatial context that 2D data alone cannot fully capture. However, acquiring detailed and realistic 3D scenes remains a major bottleneck in terms of accessibility and scalability. Conventional approaches are typically resource-expensive, labor-intensive and often not friendly for non-expert users. Powered by advances in both deep generative architectures and multimodal learning paradigms, generative models have emerged as a promising alternative. Therefore, the need for controllable, coherent, and functionally meaningful 3D scene generation from diverse input modalities defines the central problem of this dissertation: developing generative models that can synthesize realistic 3D scenes under multimodal supervision. We begin at the object level, tackling the challenging task of generating flexible and controllable 3D point clouds from a single image, especially for general and structurally complex objects such as buildings. Extending beyond individual objects, we investigate multi-object 3D scene generation with an emphasis on coherence and functional usability. Specifically, we explore how to incorporate global semantics, layout regularization, and human interaction priors into generative pipelines to ensure that the synthesized scenes are not only visually realistic but also functionally aligned with human-aware functionality.<br/><br/>

People can describe spatial scenes with language and, vice versa, create images based on linguistic descriptions. However, current systems do not even come close to matching the complexity of humans when it comes to reconstructing a scene from a given text. Even the ever-advancing development of better and better Transformer-based models has not been able to achieve this so far. This task, the automatic generation of a 3D scene based on an input text, is called text-to-3D scene generation. The key challenge, and focus of this dissertation, now relate to the following topics: (a) Analyses of how well current language models understand spatial information, how static embeddings compare, and whether they can be improved by anaphora resolution. (b) Automated resource generation for context expansion and grounding that can help in the creation of realistic scenes. (c) Creation of a VR-based text-to-3D scene system that can be used as an annotation and active-learning environment, but can also be easily extended in a modular way with additional features to solve more contexts in the future. (d) Analyze existing practices and tools for digital and virtual teaching, learning, and collaboration, as well as the conditions and strategies in the context of VR. In the first part of this work, we could show that static word embeddings do not benefit significantly from pronoun substitution. We explain this result by the loss of contextual information, the reduction in the relative occurrence of rare words, and the absence of pronouns to be substituted. But we were able to we have shown that both static and contextualizing language models appear to encode object knowledge, but require a sophisticated apparatus to retrieve it. The models themselves in combination with the measures differ greatly in terms of the amount of knowledge they allow to extract. Classifier-based variants perform significantly better than the unsupervised methods from bias research, but this is also due to overfitting. The resources generated for this evaluation are later also an important component of point three. In the second part, we present AffordanceUPT, a modularization of UPT trained on the HICO-DET dataset, which we have extended with Gibsonien/telic annotations. We then show that AffordanceUPT can effectively make the Gibsonian/telic distinction and that the model learns other correlations in the data to make such distinctions (e.g., the presence of hands in the image) that have important implications for grounding images to language. The third part first presents a VR project to support spatial annotation respectively IsoSpace. The direct spatial visualization and the immediate interaction with the 3D objects should make the labeling more intuitive and thus easier. The project will later be incorporated as part of the Semantic Scene Builder (SeSB). The project itself in turn relies on the Text2SceneVR presented here for generating spatial hypertext, which in turn is based on the VAnnotatoR. Finally, we introduce Semantic Scene Builder (SeSB), a VR-based text-to-3D scene framework using Semantic Annotation Framework (SemAF) as a scheme for annotating semantic relations. It integrates a wide range of tools and resources by utilizing SemAF and UIMA as a unified data structure to generate 3D scenes from textual descriptions and also supports annotations. When evaluating SeSB against another state-of-the-art tool, it was found that our approach not only performed better, but also allowed us to model a wider variety of scenes. The final part reviews existing practices and tools for digital and virtual teaching, learning, and collaboration, as well as the conditions and strategies needed to make the most of technological opportunities in the future.

Understanding of indoor scenes based on projection of spatial rectangles

To simulate bridge collapse realistically and assist the investigation of bridge collapse accidents, a modeling method of bridge collapse in Open Scene Graph (OSG) graph engine based on Finite element (FE) simulation is proposed. For FE simulation using 3D solid elements, the main questions of modeling, such as scene hierarchy, drawing method and texture mapping, are discussed. A study case shows the bridge model in this paper is with realistic graphics, render efficiency and simulation controllability. The 3D virtual scene simulation using the modeling method proposed in this paper has the same accuracy as FE simulation, but is more realistic and complete than FE simulation and provides a graphic support for the analysis of bridge collapse accidents.

Research on viewpoint in 3D scenes has been a cutting-edge subject in computer vision and 3D scene understanding. Calculating a high quality 3D viewpoint in virtual scenes is conducive to a more comprehensive analysis of 3D graphic structure, which can contribute to an overall understanding of virtual environments. 3D viewpoint research can also be applied to the analysis of structural relationships in virtual scenes and to exploit hidden hierarchical structures, thus it plays a role in fields such as scientific visualization computing or image-based scene modeling. Viewpoint research combined with relevant research results of human visual perception or visual psychology can be used to analyze the visual interest and focus areas of 3D objects in human vision, thus effectively improving the quality of mesh simplification and rendering efficiency, as well as providing an optimization basis for highly complex scenes. In fields such as 3D computer games, virtual reality, and landscape animation, research on high-quality viewpoints can be adopted to optimize global lighting and illumination. Additionally, practical applications can be realized through user attention analysis and aesthetic improvement strategies based on high-quality viewpoints, such as intelligent modeling and scene optimization calculations. In common fields such as 3D model retrieval, practical applications can be realized for saliency research, scientific visualization computing, and medical 3D imaging. In-depth research has been performed and extensive applications have been developed that make use of viewpoint analysis. In this paper, we introduce an optimization strategy to calculate high-quality virtual viewpoints for aesthetic images. A novel framework is proposed for modifying and optimizing the viewpoint calculation model by combining a multibranch CNN and a viewpoint correction method, thus realizing rendered images with a higher aesthetic quality. By attempting to fully integrate the visual perception with the geometric information calculation, we expect this method to achieve more comprehensive applications in many practice areas in the future, such as the realization of aesthetically based virtual camera path planning and analysis of the aesthetic characteristics of virtual environments.