Real-time holographic camera for obtaining real 3D scene hologram

Computer-assisted surveillance of complex environments is becoming more and more interesting, thanks to significant improvements in real-time signal processing. In the surveillance research field applied to public areas, crowd monitoring is very useful but presents particularly complex problems. Recognizing human bodies and tracking their movements in complex real scenes by using a sequence of images are among the most difficult tasks in computer vision. The main difficulties arise both from the high level of shape variability associated with moving human bodies, and from the complexity of environmental real scenes. A method for tracking human motion in 3D real scenes by means of Kalman filtering is presented It is based on the introduction of a suitable mathematical model for a human body moving in a 3D scene and interacting with the surrounding environment. The developed refined static and dynamic models allow the system to be accurate and robust. Due to its real-time functioning, accuracy and robustness, the method can be used in surveillance systems devoted to assure public safety.

This paper presents live and interactive webware for online learning of computer graphics concepts. A list of demos is provided. Each demo presents a concept in computer graphics by showing a 3D real world scene beside a 2D rendering scene with a list of graphics functions. Each demo allows users to interactively change the parameters and the order of execution of these graphics functions. Changing the parameters of the functions will produce the 2D rendering result from the 3D real world scene. The visual effects of user interaction will be reflected immediately in the 3D real world scene and the 2D rendering result. The webware was written by using the GL4Java library that provides native OpenGL binding for Java. Nate Robin’s well-known demos were implemented. These include translation, projection, light effect, texture mapping, and so on. New demos were also developed with pedagogical considerations in mind to emphasize the differences between model transformation and view transformation. Although the webware is designed for computer graphics learning the methodology is generic and can easily be applied to other disciplines or courses that require heavy visual presentation. This webware reflects our long-term efforts to develop web-based course material to show principles and techniques in computer science in an interactive way. We did this by having the related algorithms run live in the background and allowing students to interact with them in a web browser.

Three-dimensional (3D) light-field display has achieved a great improvement. However, the collection of dense viewpoints in the real 3D scene is still a bottleneck. Virtual views can be generated by unsupervised networks, but the quality of different views is inconsistent because networks are separately trained on each posed view. Here, a virtual view synthesis method for the 3D light-field display based on scene tower blending is presented, which can synthesize high quality virtual views with correct occlusions by blending all tower results, and dense viewpoints on 3D light-field display can be provided with smooth motion parallax. Posed views are combinatorially input into diverse unsupervised CNNs to predict respective input-view towers, and towers of the same viewpoint are fused together. All posed-view towers are blended as a scene color tower and a scene selection tower, so that 3D scene distributions at different depth planes can be accurately estimated. Blended scene towers are soft-projected to synthesize virtual views with correct occlusions. A denoising network is used to improve the image quality of final synthetic views. Experimental results demonstrate the validity of the proposed method, which shows outstanding performances under various disparities. PSNR of the virtual views are about 30 dB and SSIM is above 0.91. We believe that our view synthesis method will be helpful for future applications of the 3D light-field display.

As an important technology of digital construction, real 3D models can improve the immersion and realism of virtual reality (VR) scenes. The large amount of data for real 3D scenes requires more effective rendering methods, but the current rendering optimization methods have some defects and cannot render real 3D scenes in virtual reality. In this study, the location of the viewing frustum is predicted by a Kalman filter, and eye-tracking equipment is used to recognize the region of interest (ROI) in the scene. Finally, the real 3D model of interest in the predicted frustum is rendered first. The experimental results show that the method of this study can predict the frustrum location approximately 200 ms in advance, the prediction accuracy is approximately 87%, the scene rendering efficiency is improved by 8.3%, and the motion sickness is reduced by approximately 54.5%. These studies help promote the use of real 3D models in virtual reality and ROI recognition methods. In future work, we will further improve the prediction accuracy of viewing frustums in virtual reality and the application of eye tracking in virtual geographic scenes.

By moving a depth sensor around a room, we compute a 3D CAD model of the environment, capturing the room shape and contents such as chairs, desks, sofas, and tables. Rather than reconstructing geometry, we match, place, and align each object in the scene to thousands of CAD models of objects. In addition to the fully automatic system, the key technical contribution is a novel approach for aligning CAD models to 3D scans, based on deep reinforcement learning. This approach, which we call Learning-based ICP, outperforms prior ICP methods in the literature, by learning the best points to match and conditioning on object viewpoint. LICP learns to align using only synthetic data and does not require ground truth annotation of object pose or keypoint pair matching in real scene scans. While LICP is trained on synthetic data and without 3D real scene annotations, it outperforms both learned local deep feature matching and geometric based alignment methods in real scenes. The proposed method is evaluated on real scenes datasets of SceneNN and ScanNet as well as synthetic scenes of SUNCG. High quality results are demonstrated on a range of real world scenes, with robustness to clutter, viewpoint, and occlusion.

Three-dimensional (3D) display technology, which aims at presenting almost-realistic 3D images to the observer without any auxiliary devices, has drawn great attention from both academia and industry these years. The 3D display based on multi-layer translucent structure is a new parallax-based 3D display model. Compared with conventional parallax barrier and integrated imaging, it can effectively ensure the utilization of light energy and image resolution of the system, expand the screen depth at the same time, and display a realistic virtual 3D scene. In this paper, we implement a flat 3D display using multi-layer translucencies. Based on our prototype of flat 3D display, we further extend it to spatial 3D display using mirroring of a square pyramid, which allows observers to see virtual 3D objects from different directions in the air.

During the fieldwork of hydraulic engineering, practical engineers normally document geological information manually. Although there are some GIS-based digital tools for geology, they are not perfectly applicable to hydraulic engineering. As a result, the current work mode is ineffective, unmanageable, error-prone, and not conducive to subsequent analysis. To address this problem, we developed a digital tool which enables geological recording and quick modeling based on 3D real scenes in the field of hydropower projects. There are three modules in the surface tool: object recording, image interpretation, and field analysis. The object recording module is to mark geological points (e.g., drills and shafts), lines (e.g., faults, stratigraphic boundaries), and surfaces (e.g., slope and stocking yard) on a 3D scene and then store them in the database. The image interpretation is to interpret the 2D information in images to 3D models loaded in 3D software for further studies, such as GOCAD. The field analysis includes surface fitting, stability analysis of blocks, occurrences calculating, rock recognition, and 69/sketching. The tool is helpful for recording data, drawing geological boundaries, and building a preliminary model in the geological survey.

The appearance mode of an object, whether it appears selfluminous or reflective, depends on its luminance and its surrounding. This research aims to verify whether the appearance mode of a spherical lamp ("on" / "off") and perceived room brightness is influenced by the presentation medium: real 3D scenes (R-3D), rendered virtual 3D scenes (VR-3D) presented on a head-mounted-display (HMD) and 2D scenes presented on a regular display (D-2D). Twenty observers evaluated the lamp's appearance mode when presented in different luminance values and rated the apparent room brightness of the scene under four viewing conditions: R3D and D-2D with warm-white scene lighting, and D-2D and VR-3D with cool-white scene lighting. Threshold luminance, defined as the luminance corresponding to a 50-50 chance of perceiving a lamp as switched on, showed large observer variability, which might originate from the diversity of the observers' understanding of the lamp material and their strategy to judge the appearance mode. Respectively, threshold luminance and room brightness were significantly lower and significantly higher for the virtual reality scene than for the other conditions. However, no evidence was found that the appearance mode of a spherical lamp can relevantly predict room brightness.

Three-dimensional (3D) scene models are essential for how humans and intelligent systems perceive, reason about, and interact with the world. From digital avatars and product design to indoor environments and urban landscapes, 3D data provides semantic and spatial context that 2D data alone cannot fully capture. However, acquiring detailed and realistic 3D scenes remains a major bottleneck in terms of accessibility and scalability. Conventional approaches are typically resource-expensive, labor-intensive and often not friendly for non-expert users. Powered by advances in both deep generative architectures and multimodal learning paradigms, generative models have emerged as a promising alternative. Therefore, the need for controllable, coherent, and functionally meaningful 3D scene generation from diverse input modalities defines the central problem of this dissertation: developing generative models that can synthesize realistic 3D scenes under multimodal supervision. We begin at the object level, tackling the challenging task of generating flexible and controllable 3D point clouds from a single image, especially for general and structurally complex objects such as buildings. Extending beyond individual objects, we investigate multi-object 3D scene generation with an emphasis on coherence and functional usability. Specifically, we explore how to incorporate global semantics, layout regularization, and human interaction priors into generative pipelines to ensure that the synthesized scenes are not only visually realistic but also functionally aligned with human-aware functionality.<br/><br/>

Currently, the route planning functions in 2D/3D campus navigation systems in the market are unable to process indoor and outdoor localization information simultaneously, and the UI experiences are not optimal because they are limited by the service platforms. An ARCore-based augmented reality campus navigation system is designed in this paper in order to solve the relevant problems. Firstly, the proposed campus navigation system uses ARCore to enhance reality by presenting 3D information in real scenes. Secondly, a visual inertial ranging algorithm is proposed for real-time locating and map generating in mobile devices. Finally, rich Unity3D scripts are designed in order to enhance users’ autonomy and enjoyment during navigation experience. In this paper, indoor navigation and outdoor navigation experiments are carried out at the Lingang campus of Shanghai University of Electric Power. Compared with the AR outdoor navigation system of Gaode, the proposed AR system can achieve increased precise outdoor localization by deploying the visual inertia odometer on the mobile phone and realizes the augmented reality function of 3D information and real scene, thus enriching the user’s interactive experience. Furthermore, four groups of students have been selected for system testing and evaluation. Compared with traditional systems, such as Gaode map or Internet media, experimental results show that our system could facilitate the effectiveness and usability of learning on campus.

Abstract. 3D Real Scene is an important part of new infrastructure construction, which provides a unified spatial base for economic and social development and informatization of various departments. According to the demand of real 3D Real Scene digital construction, this paper aims to study the method of indoor and outdoor structured monomer reconstruction of city 3D Real Scene, develop digital twin platform, and strive to lead the development of 3D Real Scene. The main research contents of this paper are as follows: 1) Spatio-temporal-spectral-angular remote sensing observation system and data fusion model; 2) Rapid construction method of vector monomer 3D model of indoor and outdoor entity object; 3) 3D structural reconstruction technology of urban component level based on vector images; 4) A universal digital twins platform, LuojiaDT. The technologies we have developed have been widely used in the national 3D Real Scene construction of China, including smart city, smart transportation, cultural heritage protection, public security and police, urban underground pipe network, indoor and outdoor location service. This research will promote the continuous development of digital twins technology.

Expressing mental images visually as 3D scenes is a time-consuming challenge. Therefore, we employ natural language to facilitate the creation of virtual environments. In this paper, we present a framework, which automatically converts an arbitrary descriptive text into a representative 3D scene. Our system parses a user-written input text, extracts information using techniques from Natural Language Processing (NLP) and identifies relevant units. Based on derived object-to-object relations, our system associates every object with an appropriate 3D model and evaluates spatial dependencies of the entities. The resulting locations are combined based on adequate heuristics in order to create natural looking virtual environments. Finally, a physics engine is used to render a realistic and interactive 3D scene which enables the user to actively manipulate the stage setup.

Many advancements in the fields of holographic displays, image projection, and structured illumination rely on phase-only spatial light modulators (SLMs) to generate light fields with high dynamic range and little light loss. However, commonly used Liquid Crystal on Silicon (LCoS) SLMs are typically slow (&lt;120Hz), while SLMs which can be modulated faster tend to have limited diffraction efficiency or binary bit depth. Recently, a new class of MEMS devices, Phase Light Modulators (PLMs), have shown promise for fast (&gt;5 kHz) holographic illumination and display technologies. However, their bit depth is still limited to four-to-five bits, causing many light field modulation and optimization methods, such as Gerchberg-Saxton phase retrieval, to produce poor-quality results. In this contribution, we introduce an iterative modulation algorithm which combines a four-phase PLM level averaging method with Gerchberg-Saxton phase retrieval to enable complex field modulation while simultaneously improving the bit depth of the modulator, the PSNR of projected images, and contrast of projected images.

As an urban infrastructure project, subway engineering exhibits the characteristics of large scale, trans-regional presence, and complex environmental conditions. The three-dimensional (3D) real scene of subway engineering aims to visually present the project features by creating a virtual environment in the 3D form by integrating numerous geographical scenes and entities. However, existing methods of geometric data integration and model visualization are insufficient for handling the diverse and heterogeneous data types encountered in this process. To address these challenges, this study proposes a framework for real-scene 3D visualization management in subway engineering, establishing a technical process of multi-source data integration for organizing 3D models within the scene, and developing a visualization display strategy that incorporates octree and level of detail principles for multi-scale hierarchical dynamic loading of 3D Scene. To validate the proposed method, a prototype system is developed with Geographic Information System (GIS) and Building Information Modeling (BIM) technology and applied to an actual subway project. The results demonstrate that this method ensures effective integration and management of data, enabling the display of real scenes in an integrated 3D model in a web environment.

We introduce the first learning-based reconstructability predictor to improve view and path planning for large-scale 3D urban scene acquisition using unmanned drones. In contrast to previous heuristic approaches, our method learns a model that explicitly predicts how well a 3D urban scene will be reconstructed from a set of viewpoints. To make such a model trainable and simultaneously applicable to drone path planning, we simulate the proxy-based 3D scene reconstruction during training to set up the prediction. Specifically, the neural network we design is trained to predict the scene reconstructability as a function of the proxy geometry , a set of viewpoints, and optionally a series of scene images acquired in flight. To reconstruct a new urban scene, we first build the 3D scene proxy, then rely on the predicted reconstruction quality and uncertainty measures by our network, based off of the proxy geometry, to guide the drone path planning. We demonstrate that our data-driven reconstructability predictions are more closely correlated to the true reconstruction quality than prior heuristic measures. Further, our learned predictor can be easily integrated into existing path planners to yield improvements. Finally, we devise a new iterative view planning framework, based on the learned reconstructability, and show superior performance of the new planner when reconstructing both synthetic and real scenes.