Creating Room-Scale Interactive Mixed-Reality Worlds Using Off-the-Shelf Technologies

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Augmented Reality and Mixed Reality technologies make us feel the real world and the virtual world are mixed with each other through visual matching. However, existing mixed reality contents using these technologies have a limitation that interact only with virtual objects and cannot interact with real objects. If a dynamic real object need to be influenced by a virtual object, or vice versa, illusion of Mixed Reality can be broken by the incongruity of the interaction. If these limitations are improved, real objects, which were always static, can operate dynamically and interact with virtual objects, so becomes difficult to distinguish between real objects and virtual objects. In this paper, we propose extended interactions in a mixed reality environment that can give illusion which is difficult to distinguish between real objects and virtual objects. And to confirm this, we implement authoring environment based Hololens.

This paper proposes an interface especially tailored to create a design oriented realistic Mixed Reality (MR) workspace. An MR environment implies a natural mixture between virtual and real objects. We eplore the extension of real objects into the virtual world by having virtual objects attached to them. Also, in order to create a realistic environment, the problems of the occlusions between real and virtual objects are addresed. While providing more intuitive input methods than the conventional desktop modelers or Virtual Reality (VR) immersive systems, our prototype system enables the user to have an accurate perception of the shapes modeled. Thus, the user can recognize correctly the spacial location and real sizes of the models.

This paper describes an approach to dealing with mutual occlusions between virtual and real objects on a table-top display. Display tables use stereoscopy to make virtual content appear to exist in 3 dimensions on or above a table top. The actual image, however, lies on the physical plane of the display table. Any real physical object introduced above this plane therefore obstructs our view of the display surface and disrupts the illusion of the virtual scene. The occlusions result between real objects and the display surface, not between real objects and virtual objects. For the same reason virtual objects cannot occlude real ones. Our approach uses an additional projector located near the user's head to project those parts of virtual objects that should occlude real ones directly onto the real objects. We describe possible applications and limitations of the approach and its current implementation. Despite its limitations, we believe that the proposed approach can significantly improve interaction quality and performance for mixed reality scenarios.

Mixed reality is different from the virtual reality in that users can feel immersed in a space which is composed of not only virtual but also real objects. Thus, it is essential to realize seamless integration and interaction of the virtual and real worlds. We need depth information of the real scene to synthesize the real and virtual objects. We propose a two-stage algorithm to find smooth and precise disparity vector fields with sharp object boundaries in a stereo image pair for depth estimation. Hierarchical region-dividing disparity estimation increases the efficiency and the reliability of the estimation process, and a shape-adaptive window provides high reliability of the fields around the object boundary region. At the second stage, the vector fields are regularized with a energy model which produces smooth fields while preserving their discontinuities resulting from the object boundaries. The vector fields are used to reconstruct 3D surface of the real scene. Simulation results show that the proposed algorithm provides accurate and spatially correlated disparity vector fields in various kinds of images, and synthesized 3D models produce natural space where the virtual objects interact with the real world as if they are in the same world.

In recent years robots have dramatically improved in functionality, but as a result their designs have also become more complicated. The increase in the complexity of the tasks and the design of the robot result in a challenging and time-consuming robot development cycle. This paper identifies requirements for improving the robot development process, focusing on facilitating a safe and reliable transition process from simulation to real world tests. We propose that, by applying the concept of Mixed Reality (MR) to robot simulation, it is possible to create a realistic, synthetic environment that enables robot software developers to experiment freely and safely. The new simulation approach, named MR simulation, offers the flexibility of creating simulation environments composed of real and virtual objects that seamlessly interact with one another. This paper contributes a generic, conceptual MR framework which is demonstrated to provide a common, standardised simulation environment for experimentation involving real and virtual objects. An MR robot simulator is implemented based on the framework, integrating standardised data interfaces in its design to increase interoperability. Evaluation results demonstrate that the MR robot simulator is reusable in three different applications. It improved simulation reliability, reduced development cost, and increased efficiency.

Mixed reality technology has provided more possibilities for users to interact with both the virtual and real worlds. By integrating virtual elements with real scenes, mixed reality offers a novel immersive experience. Mixed reality also introduces complex perceptual and cognitive challenges when processing conflicting information from the virtual and real worlds. This study investigates the impact of color mismatches on response time in a mixed reality scenario. We recruited 15 participants to identify the font color of words while ignoring their meaning. The experiment included three levels of perceptual-conceptual matching. We performed a global comparison across all conditions, a within-group comparison, and a between-group comparison. Results show that in the mixed reality scenario, the processing of visual information relies on low-level physical properties (e.g., brightness and hue) and is modulated by high-level cognitive factors (e.g., conflict processing and expectation). These findings indicate that visual mismatches in mixed reality can significantly affect response time, particularly when perceptual salience is low and different from cognitive expectations. The results have implications for the design of mixed reality interfaces in complex environments where users must process information from the virtual and real worlds, such as head-up displays, warehouse management systems, and industrial instructional overlays.

Real Time Distributed Content Rendering Technique based on Agent-mediator Communication Framework for Multi-display Systems

This chapter introduces the field of Mixed Reality that merges the real and virtual worlds seamlessly. To develop a Mixed Reality system, two fundamental problems should be addressed: tracking and user interface. Tracking in Mixed Reality is defined as the task of estimating the user's viewing pose. The goal of tracking is to achieve an accurate registration between real and virtual objects. In order to allow the user to interact e±ciently with the Mixed Reality scene, interaction techniques should be designed. A wide range of Mixed Reality applications, such as medical and entertainment applications, have been identified. We will focus on the new trends and applications in the area of Mixed Reality.

The perceptual behavior of consumers on a product displayed in the market has a vital role in analyzing the importance given to that product. Therefore, various strategies have been developed to understand this consumer behavior in the selection of products. Immersive technologies like virtual, augmented, and mixed reality are among them. With the foremost feature of immersion in the virtual world and interaction of users with virtual objects, virtual reality, and augmented reality have unlocked their potential in research and a user-friendly tool for analyzing consumer behavior. In addition to these technologies, mixed reality also has a significant role in investigating consumer behavior. Studies on immersive technologies in food applications are vast, hence this review focuses on the applications of virtual, augmented, and mixed reality in the food selection behavior of consumers. The behavioral studies are elicited to develop new products based on consumer needs, to understand the shopping behavior in supermarkets for real-time usage, and to know the influence of emotions in a selection of products. The findings suggest that virtual, augmented, and mixed reality induce immersion of the users in food selection behavioral studies. Information on the technological advancements in the tools used for bringing immersion and interaction are discussed for its futuristic applications in food. Though immersive technology gives users a realistic virtual environment experience, its application in food systems is in the budding stage. More research on human response studies would contribute to its innovative and inevitable application in the future.

This study aimed to analyze the impact of design classes based on mixed reality (MR) on the learning experiences and outcomes of elementary school students and to explore the characteristics of human-technology interactions in the MR environment. The research was conducted with 24 sixth-grade students at S Elementary School in Seoul, allowing learners to experience creative learning in an immersive environment where virtual and real worlds are combined through MR technology. The results show that MR-based design education significantly enhanced students' spatial understanding and problem-solving abilities, and positively influenced collaborative learning and creative interaction. Additionally, human-technology interaction in the MR environment provided students with the experience of handling both virtual and physical objects simultaneously, contributing to the improvement of digital literacy and collaborative problem-solving skills. These findings suggest the potential of MR technology as an innovative educational method suitable for the digital age, highlighting its applicability in future educational settings.

One of the main problems of mixed reality devices is the physically correct representation of the luminance distribution for virtual objects and their shadows in the real world. In other words, restoring the correct distribution of scene luminance is one of the key parameters that allows solving the problem of correct interaction between the virtual and real worlds. The paper proposes methods for restoring the parameters of light sources. Also, the work is aimed at the creation and formation of criteria for the quality of visual perception, allowing to evaluate the synthesized image of mixed reality and to decide how natural it is from the point of view of the observer. In this article, surveys were used to create datasets using realistic software tools. In work the neural network is trained to recognize those areas of the image that do not fit into the environment and to classify this image as a class that causes visual discomfort. As criteria for the quality of visual perception, it is proposed to use estimates of the mismatch of the parameters of shadows cast by virtual objects and the distribution of luminance over these objects in the images of scenes containing models of "real" and "virtual" objects. The level of misalignment is estimated concerning the true lighting conditions of the real world. In this work, criteria for the quality of visual perception were formed and a neural network was trained, which makes it possible to decide and analyze the quality of a synthesized mixed reality image.

This paper proposes a Blended Reality Agent called "BReA" that can exist in both the real and virtual world and has communicative advantages that is more than simply the sum of those of a robotic agent and an on-screen agent. BReA can seamlessly transfer between the real world and virtual world and users can communicate with BReA without interruption during the transference. We conducted a field test and showed that users can recognize when it points from the virtual world to a real world object after it moves from the real world to virtual world. This result supports a hypothesis that communicating with BReA helps users recognize the link between real world and virtual world objects and virtual world information.

Mixed Reality applications create highly immersive experiences by integrating real world objects into the virtual world and thus expanding the virtual experience by a tactile perception. Interaction with mixed reality worlds becomes more natural if the user can interact using only his hands. Hence, the question arises whether the experience in mixed reality applications can be increased if the user’s hands and body are integrated and virtually represented as well. Hence, as a proof of concept, we developed a mixed reality application in which every physical object has a respective virtual counterpart. The major concept of the development is to experience height while walking over a wooden plank and to solve an interaction task afterwards. We present an empirical study that examines the effect of a full body vs a hands vs no representation on the experience on presence and the physiological arousal. Results indicate that integrating a virtual body increases the experience of presence as well as the physiological arousal.

In the mysterious realm full of emerging technologies, there is a fantastic role played by Mixed Reality techniques. The blending of real and virtual worlds that allows individualized user control opportunities of the environment around user remarks the unique feature of Mixed Reality. The current work shows the effort of many researchers who have tried to explore the field of Mixed Reality in their own planned ways for the creation of Mixed Reality smart classroom. The objective is to introduce, how Mixed Reality device can act as real-time analytical tool for the teacher in a classroom to enhance the learning process of the students. The first part introduces the advent of Mixed Reality with some defined theories that are widely accepted and a glimpse of approaches done by many researchers. Later, an overview of an attempt to create a Mixed Reality classroom, two different hypotheses incorporating the same Mixed Reality tool called Lumilo were experimented with the assistance of AIED and ITS systems. Followed by the results of the hypotheses, aiming one of them to be employed in the classroom environment where both teachers and students were most benefitted. This type of environment resulted in better modes of education when compared to the existing approaches of Mixed Reality classroom.