Exploring Interactions Specific to Mixed Reality 3D Modeling Systems

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Augmented Reality and Mixed Reality technologies make us feel the real world and the virtual world are mixed with each other through visual matching. However, existing mixed reality contents using these technologies have a limitation that interact only with virtual objects and cannot interact with real objects. If a dynamic real object need to be influenced by a virtual object, or vice versa, illusion of Mixed Reality can be broken by the incongruity of the interaction. If these limitations are improved, real objects, which were always static, can operate dynamically and interact with virtual objects, so becomes difficult to distinguish between real objects and virtual objects. In this paper, we propose extended interactions in a mixed reality environment that can give illusion which is difficult to distinguish between real objects and virtual objects. And to confirm this, we implement authoring environment based Hololens.

This paper describes an approach to dealing with mutual occlusions between virtual and real objects on a table-top display. Display tables use stereoscopy to make virtual content appear to exist in 3 dimensions on or above a table top. The actual image, however, lies on the physical plane of the display table. Any real physical object introduced above this plane therefore obstructs our view of the display surface and disrupts the illusion of the virtual scene. The occlusions result between real objects and the display surface, not between real objects and virtual objects. For the same reason virtual objects cannot occlude real ones. Our approach uses an additional projector located near the user's head to project those parts of virtual objects that should occlude real ones directly onto the real objects. We describe possible applications and limitations of the approach and its current implementation. Despite its limitations, we believe that the proposed approach can significantly improve interaction quality and performance for mixed reality scenarios.

Hand-tracking technologies allow us to use our own hands to interact with real and virtual objects in Augmented Reality (AR) environments. This enables us to explore the interplay between hand-visualizations and hand-object interactions. We present a user study that examines the effect of different hand visualizations (invisible, transparent, opaque) on manipulation performance when interacting with real and virtual objects. For this, we implemented video-see-through (VST) AR-based virtual building blocks and hot wire tasks with real one-to-one counterparts that require participants to use gross and fine motor hand movements. To evaluate manipulation performance, we considered three measures: task completion time, number of collisions (hot wire task), and percentage of object displacement (building block task). Additionally, we explored the sense of agency and subjective impressions (preference, ease of interaction, successful and awkwardness) evoked by the different hand-visualizations. The results show that (1) manipulation performance is significantly higher when interacting with real objects compared to virtual ones, (2) invisible hands lead to fewer errors, higher agency, higher perceived success and ease of interaction during fine manipulation tasks with real objects, and (3) having some visualization of the virtual hands (transparent or opaque) overlayed on the real hands is preferred when manipulating virtual objects even when there are no significant performance improvements. Our empirical findings about the differences when interacting with real and virtual objects can aid hand visualization choices for manipulation tasks in AR.

The perceptual behavior of consumers on a product displayed in the market has a vital role in analyzing the importance given to that product. Therefore, various strategies have been developed to understand this consumer behavior in the selection of products. Immersive technologies like virtual, augmented, and mixed reality are among them. With the foremost feature of immersion in the virtual world and interaction of users with virtual objects, virtual reality, and augmented reality have unlocked their potential in research and a user-friendly tool for analyzing consumer behavior. In addition to these technologies, mixed reality also has a significant role in investigating consumer behavior. Studies on immersive technologies in food applications are vast, hence this review focuses on the applications of virtual, augmented, and mixed reality in the food selection behavior of consumers. The behavioral studies are elicited to develop new products based on consumer needs, to understand the shopping behavior in supermarkets for real-time usage, and to know the influence of emotions in a selection of products. The findings suggest that virtual, augmented, and mixed reality induce immersion of the users in food selection behavioral studies. Information on the technological advancements in the tools used for bringing immersion and interaction are discussed for its futuristic applications in food. Though immersive technology gives users a realistic virtual environment experience, its application in food systems is in the budding stage. More research on human response studies would contribute to its innovative and inevitable application in the future.

In recent years robots have dramatically improved in functionality, but as a result their designs have also become more complicated. The increase in the complexity of the tasks and the design of the robot result in a challenging and time-consuming robot development cycle. This paper identifies requirements for improving the robot development process, focusing on facilitating a safe and reliable transition process from simulation to real world tests. We propose that, by applying the concept of Mixed Reality (MR) to robot simulation, it is possible to create a realistic, synthetic environment that enables robot software developers to experiment freely and safely. The new simulation approach, named MR simulation, offers the flexibility of creating simulation environments composed of real and virtual objects that seamlessly interact with one another. This paper contributes a generic, conceptual MR framework which is demonstrated to provide a common, standardised simulation environment for experimentation involving real and virtual objects. An MR robot simulator is implemented based on the framework, integrating standardised data interfaces in its design to increase interoperability. Evaluation results demonstrate that the MR robot simulator is reusable in three different applications. It improved simulation reliability, reduced development cost, and increased efficiency.

Mixed reality is different from the virtual reality in that users can feel immersed in a space which is composed of not only virtual but also real objects. Thus, it is essential to realize seamless integration and interaction of the virtual and real worlds. We need depth information of the real scene to synthesize the real and virtual objects. We propose a two-stage algorithm to find smooth and precise disparity vector fields with sharp object boundaries in a stereo image pair for depth estimation. Hierarchical region-dividing disparity estimation increases the efficiency and the reliability of the estimation process, and a shape-adaptive window provides high reliability of the fields around the object boundary region. At the second stage, the vector fields are regularized with a energy model which produces smooth fields while preserving their discontinuities resulting from the object boundaries. The vector fields are used to reconstruct 3D surface of the real scene. Simulation results show that the proposed algorithm provides accurate and spatially correlated disparity vector fields in various kinds of images, and synthesized 3D models produce natural space where the virtual objects interact with the real world as if they are in the same world.

Mixed-Reality (MR) represents a combination of real and virtual worlds. Off-the-shelf solutions already exist for quite some time which enable the creation of MR worlds where physical and digital objects co-exist and interact in real time. However, little is still known regarding methodological and technical challenges involved in the development of room scale MR worlds with the use of high-end technologies, especially when considering real-object representation. This paper reports hands-on experiences in creating a room-scale MR world, using widespread, off-the-shelf object-tracking technologies and 3D modelling techniques, which enable free user movement, accurate real-object representation in the virtual world, as well as interactivity between real and virtual objects.

Augmented reality (AR) is anticipated for use in medical practice. However, virtual objects cannot be overlaid exactly on real objects using theoretical data. Therefore, this study is intended to determine the correction method to overlay a virtual object with real object accurately. Experimental results confirmed that differences of perceived depth position between real and virtual objects can be made negligible (within 1 mm on average) by correcting sizes of virtual objects such that they can be seen identically to the real object.

Visual coherence between real and virtual objects is important in mixed reality (MR), and illumination consistency is one of the key aspects to achieve coherence. Apart from matching the illumination of the virtual objects with the real environments, the change of illumination on the real scenes produced by the inserted virtual objects should also be considered but is difficult to compute in real-time due to the heavy computation demands of global illumination. In this work, we propose delta path tracing (DPT), which only computes the radiance blocked by the virtual objects from the light sources at the primary hit points of Monte Carlo path tracing, then combines the blocked radiance and multi-bounce indirect illumination with the image of the real scene. Multiple importance sampling (MIS) between BRDF and environment map is performed to handle all-frequency environment maps captured by a panorama camera. Compared to conventional differential rendering methods, our method can remarkably reduce the number of times required to access the environment map and avoid rendering scenes twice. Therefore, the performance can be significantly improved. We implement our method using hardware-accelerated ray tracing on modern GPUs, and the results demonstrate that our method can render global illumination at real-time frame rates and produce plausible visual coherence between real and virtual objects in MR environments.

Though virtual reality (VR) has been advanced to certain levels of maturity in recent years, the general public, especially the population of the blind and visually impaired (BVI), still cannot enjoy the benefit provided by VR. Current VR accessibility applications have been developed either on expensive head-mounted displays or with extra accessories and mechanisms, which are either not accessible or inconvenient for BVI individuals. In this paper, we present a mobile VR app that enables BVI users to access a virtual environment on an iPhone in order to build their skills of perception and recognition of the virtual environment and the virtual objects in the environment. The app uses the iPhone on a selfie stick to simulate a long cane in VR, and applies Augmented Reality (AR) techniques to track the iPhone’s real-time poses in an empty space of the real world, which is then synchronized to the long cane in the VR environment. Due to the use of mixed reality (the integration of VR & AR), we call it the Mixed Reality cane (MR Cane), which provides BVI users auditory and vibrotactile feedback whenever the virtual cane comes in contact with objects in VR. Thus, the MR Cane allows BVI individuals to interact with the virtual objects and identify approximate sizes and locations of the objects in the virtual environment. We performed preliminary user studies with blind-folded participants to investigate the effectiveness of the proposed mobile approach and the results indicate that the proposed MR Cane could be effective to help BVI individuals in understanding the interaction with virtual objects and exploring 3D virtual environments. The MR Cane concept can be extended to new applications of navigation, training and entertainment for BVI individuals without more significant efforts.

Augmented Reality (AR) is the combination of digital information and user's target in real world. The handheld devices like Smartphone have all the components that are necessary to use an augmented reality application, where the virtual three dimensional objects imported on real world environment. In AR based mobile applications utilizing an image target trackers are growing. In these continue growing trends can lead the applications to require more interaction technique interfaces, which will attract the user attentions (e.g. hand interaction). In this proposes work a new approach used to dynamic nature hand interaction on Virtual Reality (VR) objects through the mobile based AR application. Virtual link and buttons are placed on the image target surface and interact by the human 3-D hand. Hand detection algorithm used for fast detecting and track the hand on virtual objects. By this proposed system the user can directly change the behavior and position of the virtual object by human hand. Virtual links and buttons are allowing the users to view other virtual objects on same virtual space. Filter data technique by reducing redundancies and creating layout for remaining information on virtual objects for change the behavior or functions of the virtual interface objects.

The mixed reality makes the user's realism strong or weak, depending on the fusion effect of the virtual object and the real scene, and the key factor affecting the virtual-real fusion effect is whether the virtual and real object in the synthetic scene has occlusion consistency. In this paper, a method is proposed that utilizes color and depth information provided by an RGB-D camera to improved occlusion. In this method, based on the GPU, extracting the ROI area where the virtual object is located (the area where the virtual-real occlusion occurs),then introducing the real scence color image as the guide image to modified joint bilateral filtering the depth image to repair the wrong edge information and "black hole" of the depth image;pixel-by-pixel point to determine the depth relationship between the virtual object and real object, to achieve virtual-real occlusion rendering blend;for the still existing "sawtooth artifacts" using delay-coloring only fuzzy local boundary algorithm, to achieve accurate and smoothing edges virtual-real occlusion to enhance visual effects.

One of the main problems of monoscopic video see-through augmented reality (AR) is the lack of reliable depth information. This makes it difficult to correctly represent complex spatial interactions between real and virtual objects, e.g., when rendering shadows. The most obvious graphical artifact is the incorrect display of the occlusion of virtual models by real objects. Since the graphical models are rendered opaquely over the camera image, they always appear to occlude all objects in the real environment, regardless of the actual spatial relationship. In this paper, we propose to utilize a new type of hardware in order to solve some of the basic challenges of AR rendering. We introduce a depth-of-flight range sensor into AR, which produces a 2D map of the distances to real objects in the environment. The distance map is registered with high resolution color images delivered by a digital video camera. When displaying the virtual models in AR, the distance map is used in order to decide whether the camera image or the virtual object is visible at any position. This way, the occlusion of virtual models by real objects can be correctly represented. Preliminary results obtained with our approach show that a useful occlusion handling based on time-of-flight range data is possible.

Outdoor rendering is an attractive topic in computer graphics. In this paper our main concern is to reveal the interaction between sky color and virtual objects in Mixed Reality environments. Although registration and tracking are two of the main issues in building effective Augmented Reality (AR) systems the creation of more realistic virtual objects indistinguishable from their real-world counterparts is our target which is currently the ultimate goal in AR. Two classes of sky color generation are employed to reveal the outdoor-element interaction. Virtual Sky Modelling (VSM) based on the Perez Model is capable of generating the sky color in a specific location, date and time. The second technique is to generate a virtual model based on the real image of the sky which is called in this paper Real Sky Modelling (RSM). Subsequently, preprocessing of the sky color bleeding is based on the radiosity technique to give the sky color effect to the virtual objects as well as the real ones. Through designing a test AR set-up and applying software and hardware the goal of a robust generation of realistic virtual objects with effect of sky color is achieved.