Language-Guided Compositional 3D Scene Generation via Fine-Grained Part-level 3D Representation

The evolution of generative models has been quite rapid, and this has greatly impacted the field of 3D scene generation. It was previously impossible to build such detailed 3D scenes automatically and with relative simplicity because the process demanded considerable human input alongside significant computational power. At the same time, it could be a labor-intensive task, especially when a large number of scenes needed to be generated. New generative synthesis methods such as diffusion models and GANs have made it much easier and more efficient to synthesize high-quality 3D scenes. Of these, the diffusion models have turned out to excel at crafting diverse and highly complex multi-object 3D scenes that are photorealistic and provide a new record in scene generation. Such technologies have enabled new opportunities across industries such as VR, augmented reality, gaming, robotics, and automation, where approximate and more realistic 3D models are increasingly becoming useful. The diffusion models, notably, show comparatively high performance in creating complex spatial patterns and a high level of detail, which is beneficial for realistic applications. This paper discusses the basics, uses, limitations, and innovations of current generative models, with an emphasis on diffusion models, with a view to applying them mainly to 3D scene generation. Thus, this review highlights the possibilities and oversights of diffusion models to demonstrate the possibilities of the future of automated 3D content creation across several technological domains.

Three-dimensional (3D) scene models are essential for how humans and intelligent systems perceive, reason about, and interact with the world. From digital avatars and product design to indoor environments and urban landscapes, 3D data provides semantic and spatial context that 2D data alone cannot fully capture. However, acquiring detailed and realistic 3D scenes remains a major bottleneck in terms of accessibility and scalability. Conventional approaches are typically resource-expensive, labor-intensive and often not friendly for non-expert users. Powered by advances in both deep generative architectures and multimodal learning paradigms, generative models have emerged as a promising alternative. Therefore, the need for controllable, coherent, and functionally meaningful 3D scene generation from diverse input modalities defines the central problem of this dissertation: developing generative models that can synthesize realistic 3D scenes under multimodal supervision. We begin at the object level, tackling the challenging task of generating flexible and controllable 3D point clouds from a single image, especially for general and structurally complex objects such as buildings. Extending beyond individual objects, we investigate multi-object 3D scene generation with an emphasis on coherence and functional usability. Specifically, we explore how to incorporate global semantics, layout regularization, and human interaction priors into generative pipelines to ensure that the synthesized scenes are not only visually realistic but also functionally aligned with human-aware functionality.<br/><br/>

Abstract3D scene understanding and generation are to reconstruct the layout of the scene and each object from an RGB image, estimate its semantic type in 3D space and generate a 3D scene. At present, the 3D scene generation algorithm based on deep learning mainly recovers the 3D scene from a single image. Due to the complexity of the real environment, the information provided by a single image is limited, and there are problems such as the lack of single-view information and the occlusion of objects in the scene. In response to the above problems, we propose a 3D scene generation framework SGMT, which realizes multi-view position information fusion and reconstructs the 3D scene from multi-view video time series data to compensate for the missing object position in existing methods. We demonstrated the effectiveness of multi-view scene generation of SGMT on the UrbanScene3D and SUNRGBD dataset and studied the influence of SGCN and joint fine-tuning. In addition, we further explored the transfer ability of the SGMT between datasets and discussed future improvements.Keywords3D scene generationMulti-view fusionMulti-view time series data

Due to overwhelming use of 3D models in video games and virtual environments, there is a growing interest in 3D scene generation, scene understanding and 3D model retrieval. In this paper, we introduce a data-driven 3D scene generation approach from a Maximum Entropy (MaxEnt) model selection perspective. Using this model selection criterion, new scenes can be sampled by matching a set of contextual constraints that are extracted from training and synthesized scenes. Starting from a set of random synthesized configurations of objects in 3D, the MaxEnt distribution is iteratively sampled (using Metropolis sampling) and updated until the constraints between training and synthesized scenes match, indicating the generation of plausible synthesized 3D scenes. To illustrate the proposed methodology, we use 3D training desk scenes that are all composed of seven predefined objects with different position, scale and orientation arrangements. After applying the MaxEnt framework, the synthesized scenes show that the proposed strategy can generate reasonably similar scenes to the training examples without any human supervision during sampling. We would like to mention, however, that such an approach is not limited to desk scene generation as described here and can be extended to any 3D scene generation problem.

Recently, Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have shown promise in instruction following and image understanding. While these models are powerful, they have not yet been developed to comprehend the more challenging 3D geometric and physical scenes, especially when it comes to the sparse outdoor LiDAR data. In this paper, we introduce LiDAR-LLM, which takes raw LiDAR data as input and harnesses the remarkable reasoning capabilities of LLMs to gain a comprehensive understanding of outdoor 3D scenes. The central insight of our LiDAR-LLM is the reformulation of 3D outdoor scene cognition as a language modeling problem, encompassing tasks such as 3D captioning, 3D grounding, 3D question answering, etc. Specifically, due to the scarcity of 3D LiDAR-text pairing data, we introduce a three-stage training strategy and generate relevant datasets, progressively aligning the 3D modality with the language embedding of LLM. Furthermore, we design a Position-Aware Transformer (PAT) to connect the 3D encoder with the LLM, which effectively bridges the modality gap and enhances the LLM's spatial orientation comprehension of visual features. Our experiments demonstrate that LiDAR-LLM effectively comprehends a wide range of instructions related to 3D scenes, achieving a 40.9 BLEU-1 score on the 3D captioning dataset, a Grounded Captioning accuracy of 63.1%, and a BEV mIoU of 14.3%.

3D scene generation for zero-shot learning using ChatGPT guided language prompts

In a complex 3D scene, which typically contains multiple models with distinct feature differences, determining how to set appropriate simplification rates for each model so that the 3D scene maintains its characteristics well while achieving the desired simplification rate is a challenging task. To address this, this paper proposes a 3D scene simplification algorithm that integrates neural networks to accomplish this task. The algorithm is divided into two stages: the first stage extracts local and global features of each model in the 3D scene to form feature vectors, and the second stage combines neural networks to construct a multi-layer perceptron that predicts the simplification labels for each model in the 3D scene and calculates the simplification weights for each model based on these labels. Experiments have proven that this algorithm not only maintains a good appearance of the 3D scene but also has a smaller overall simplification error compared to the traditional QEM algorithm.

The current generation of large language models (LLMs) has limited chemical knowledge. Recently, it has been shown that these LLMs can learn and predict chemical properties through fine-tuning. Using natural language to train machine learning models opens doors to a wider chemical audience, as field-specific featurization techniques can be omitted. In this work, we explore the potential and limitations of this approach. We studied the performance of fine-tuning three open-source LLMs (GPT-J-6B, Llama-3.1-8B, and Mistral-7B) for a range of different chemical questions. We benchmark their performances against “traditional” machine learning models and find that, in most cases, the fine-tuning approach is superior for a simple classification problem. Depending on the size of the dataset and the type of questions, we also successfully address more sophisticated problems. The most important conclusions of this work are that, for all datasets considered, their conversion into an LLM fine-tuning training set is straightforward and that fine-tuning with even relatively small datasets leads to predictive models. These results suggest that the systematic use of LLMs to guide experiments and simulations will be a powerful technique in any research study, significantly reducing unnecessary experiments or computations.

Scene creation in video games is an iterative process that transitions from initial ideas to 2D visuals and finally to 3D scenes, requiring collaborative teamwork among gameplay and environment designers, 3D modelers, and technical artists. While advances in generative methods have enabled the automation of scene creation, these end-to-end approaches often do not align with the typical workflow. This paper presents SceneWeaver, a 3D scene creation system that utilizes a multi-agent collaborative framework with large language models (LLMs) assigned to manage text parsing, floorplan design, object selection, and scene composition. An interactive visual interface with step-by-step previsualization to enhance human-AI communication and collaboration throughout the scene creation process. SceneWeaver effectively enhances user engagement and improves process controllability in 3D scene creation, as confirmed by performance evaluations and user studies.

This paper presents developing and implementing an AI-driven WebTV system designed to automate the video content creation pipeline, from conceptualization to broadcasting. Leveraging open-source models such as Zeroscope for text-to-video generation and MusicGen for music synthesis, the system integrates large language models (LLMs) to create a seamless, modular production architecture. The research explores the challenges of combining different AI components into a cohesive framework, highlighting issues such as real-time processing, video quality optimization, and model interoperability. The findings demonstrate the system’s ability to generate coherent, good-quality video and audio content autonomously, significantly reducing the need for human intervention in traditional video production workflows. Although the AI-driven WebTV system illustrates the potential for scalable, automated media production, limitations in scene complexity, frame interpolation, and content consistency are identified. Future work is suggested to enhance the system’s scalability, real-time adaptability, and ethical content generation. This research underscores the transformative potential of AI in media production, offering a foundation for future exploration into fully autonomous digital content creation.

A recent revolution in miniaturised sensor technology has provided markets with novel hyperspectral imagers operating in the frame format principle. In the case of unmanned aerial vehicle (UAV) based remote sensing, the frame format technology is highly attractive in comparison to the commonly utilised pushbroom scanning technology, because it offers better stability and the possibility to capture stereoscopic data sets, bringing an opportunity for 3D hyperspectral object reconstruction. Tuneable filters are one of the approaches for capturing multi- or hyperspectral frame images. The individual bands are not aligned when operating a sensor based on tuneable filters from a mobile platform, such as UAV, because the full spectrum recording is carried out in the time-sequential principle. The objective of this investigation was to study the aspects of band registration of an imager based on tuneable filters and to develop a rigorous and efficient approach for band registration in complex 3D scenes, such as forests. The method first determines the orientations of selected reference bands and reconstructs the 3D scene using structure-from-motion and dense image matching technologies. The bands, without orientation, are then matched to the oriented bands accounting the 3D scene to provide exterior orientations, and afterwards, hyperspectral orthomosaics, or hyperspectral point clouds, are calculated. The uncertainty aspects of the novel approach were studied. An empirical assessment was carried out in a forested environment using hyperspectral images captured with a hyperspectral 2D frame format camera, based on a tuneable Fabry-Pérot interferometer (FPI) on board a multicopter and supported by a high spatial resolution consumer colour camera. A theoretical assessment showed that the method was capable of providing band registration accuracy better than 0.5-pixel size. The empirical assessment proved the performance and showed that, with the novel method, most parts of the band misalignments were less than the pixel size. Furthermore, it was shown that the performance of the band alignment was dependent on the spatial distance from the reference band.

Computer-assisted surveillance of complex environments is becoming more and more interesting, thanks to significant improvements in real-time signal processing. In the surveillance research field applied to public areas, crowd monitoring is very useful but presents particularly complex problems. Recognizing human bodies and tracking their movements in complex real scenes by using a sequence of images are among the most difficult tasks in computer vision. The main difficulties arise both from the high level of shape variability associated with moving human bodies, and from the complexity of environmental real scenes. A method for tracking human motion in 3D real scenes by means of Kalman filtering is presented It is based on the introduction of a suitable mathematical model for a human body moving in a 3D scene and interacting with the surrounding environment. The developed refined static and dynamic models allow the system to be accurate and robust. Due to its real-time functioning, accuracy and robustness, the method can be used in surveillance systems devoted to assure public safety.

Lift3Dreamer: Boosting Text-Driven Novel View Synthesis via Lifted 3D Inpainting Model from Single Images.

3D immersive scene generation is a challenging yet critical task in computer vision and graphics. A desired virtual 3D scene should 1) exhibit omnidirectional view consistency, and 2) allow for large-range exploration in complex scene hierarchies. Existing methods either rely on successive scene expansion via inpainting or employ panorama representation to represent large FOV scene environments. However, the generated scene suffers from semantic drift during expansion and is unable to handle occlusion among scene hierarchies. To tackle these challenges, we introduce LayerPano3D, a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt. Our key insight is to decompose a reference 2D panorama into multiple layers at different depth levels, where each layer reveals the unseen space from the reference views via diffusion prior. LayerPano3D comprises multiple dedicated designs: 1) We introduce a new panorama dataset Upright360 , comprising 9k high-quality and upright panorama images, and finetune the advanced Flux model on Upright360 for high-quality, upright and consistent panorama generation related tasks. 2) We pioneer the Layered 3D Panorama as underlying representation to manage complex scene hierarchies and lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes with unconstrained viewing paths. Extensive experiments demonstrate that our framework generates state-of-the-art 3D panoramic scene in both full view consistency and immersive exploratory experience. We believe that LayerPano3D holds promise for advancing 3D panoramic scene creation with numerous applications. More examples please visit our webpage: ys-imtech.github.io/projects/LayerPano3D/.

Implicit Neural Representations (INR) use multilayer perceptrons to represent high-frequency functions in low-dimensional problem domains. Recently these representations achieved state-of-the-art results on tasks related to complex 3D objects and scenes. A core problem is the representation of highly detailed signals, which is tackled using networks with periodic activation functions (SIRENs) or applying Fourier mappings to the input. This work analyzes the connection between the two methods and shows that a Fourier mapped perceptron is structurally like one hidden layer SIREN. Furthermore, we identify the relationship between the previously proposed Fourier mapping and the general d-dimensional Fourier series, leading to an integer lattice mapping. Moreover, we modify a progressive training strategy to work on arbitrary Fourier mappings and show that it improves the generalization of the interpolation task. Lastly, we compare the different mappings on the image regression and novel view synthesis tasks. We confirm the previous finding that the main contributor to the mapping performance is the size of the embedding and standard deviation of its elements.