Designing Mixed Reality: Principles, Projects and Practice

We present an optimization-based approach that automatically adapts Mixed Reality (MR) interfaces to different physical environments. Current MR layouts, including the position and scale of virtual interface elements, need to be manually adapted by users whenever they move between environments, and whenever they switch tasks. This process is tedious and time consuming, and arguably needs to be automated for MR systems to be beneficial for end users. We contribute an approach that formulates this challenge as a combinatorial optimization problem and automatically decides the placement of virtual interface elements in new environments. To achieve this, we exploit the semantic association between the virtual interface elements and physical objects in an environment. Our optimization furthermore considers the utility of elements for users’ current task, layout factors, and spatio-temporal consistency to previous layouts. All those factors are combined in a single linear program, which is used to adapt the layout of MR interfaces in real time. We demonstrate a set of application scenarios, showcasing the versatility and applicability of our approach. Finally, we show that compared to a naive adaptive baseline approach that does not take semantic associations into account, our approach decreased the number of manual interface adaptations by 33%.

Interacting with real-world objects in Mixed Reality (MR) often proves difficult when they are crowded, distant, or partially occluded, hindering straightforward selection and manipulation. We observe that these difficulties stem from performing interaction directly on physical objects, where input is tightly coupled to their physical constraints. Our key insight is to decouple interaction from these constraints by introducing proxies-abstract representations of real-world objects. We embody this concept in Reality Proxy, a system that seamlessly shifts interaction targets from physical objects to their proxies during selection. Beyond facilitating basic selection, Reality Proxy uses AI to enrich proxies with semantic attributes and hierarchical spatial relationships of their corresponding physical objects, enabling novel and previously cumbersome interactions in MR - such as skimming, attribute-based filtering, navigating nested groups, and complex multi object selections - all without requiring new gestures or menu systems. We demonstrate Reality Proxy's versatility across diverse scenarios, including office information retrieval, large-scale spatial navigation, and multi-drone control. An expert evaluation suggests the system's utility and usability, suggesting that proxy-based abstractions offer a powerful and generalizable interaction paradigm for future MR systems.

Mixed Reality (MR) technologies provide users with an environment that incorporates virtual objects and metadata into their physical surroundings. This opens up exciting new possibilities for applications in various domains including: education, training, healthcare, and Computer Supported Cooperative Work (CSCW). Recognizing physical objects in the environment provides contextual clues that can improve user task performance and influence effective cognitive load for complex tasks. However, this facet of MR is still relatively nascent because of hardware limitations. Presently, MR headsets can only detect physical surfaces; they can not distinguish between physical artifacts that a user might need in their work environment. Machine Learning (ML) based object recognition solutions can help overcome this limitation. However, current commercially available MR devices have insufficient computational capabilities and are unable to support the state-of-the-art ML based object recognition solutions. To address these challenges, we describe a novel approach and the corresponding framework by introducing a hardware level separation between the two tasks of object recognition and rendering virtual components. Our approach leverages ML based object recognition, MR, and a portable IoT edge computing platform to support contextual awareness. We describe the implementation of this approach and provide a case study. The prototype implementation uses an edge device to process and recognize objects to get information about the user’s physical environment and provides the user with situational awareness through the MR device. Our case study uses a Christmas tree decoration activity to demonstrate the ability of the proposed framework to support assembly-type tasks. The case study establishes the applicability of the proposed approach to a variety of problem domains. Object recognition and context aware capabilities, combined with virtual indicators and instructions, provide a solution that increases usability, helps reduce human error, and improves the overall task performance.

The last decade has seen a significant surge in the popularity of virtual reality in countless fields and it keeps on spreading into more areas of application. While Virtual Reality (VR) is already being used in various educational settings, such as medicine and teaching, the technology is constantly being enhanced and augmented into Mixed Reality (MR) approaches. Especially in the training of certain skills, mixed reality provides numerous new training perspectives. The specific domain of special education is no exception to that fact and potential applications of VR are increasingly being explored. The aim of this work is to develop a VR trainer tor the training of special educators, in order for them to acquire new skills in specific teaching techniques. All aspects of this work are conducted in collaboration with Reykjavik University, where previous work on the topic was conducted in 2017. The goal is to improve on the existing trainer by adding new interaction modalities, such as voice input and virtual touch. In addition to that, the state of the art is pushed even further by introducing Augmented Virtuality (AV) through physical objects in a second version of this trainer. The goal of implementing a new simulator for training special education teachers is successfully achieved, as witnessed during two user tests with students and teachers from the field of special education. In the future, this VR trainer will continue to be used at Reykjavik University, superseding the previous one. The goal of introducing AV through physical objects is achieved through a viable technical solution that still needs to be simplified in its deployment before extensive user testing.

Mixed reality offers new potentials for social interaction experiences with virtual agents. In addition, it can be used to experiment with the design of physical robots. However, while previous studies have investigated comfortable social distances between humans and artificial agents in real and virtual environments, there is little data with regards to mixed reality environments. In this paper, we conducted an experiment in which participants were asked to walk up to an agent to ask a question, in order to investigate the social distances maintained, as well as the subject's experience of the interaction. We manipulated both the embodiment of the agent (robot vs. human and virtual vs. physical) as well as closed vs. open posture of the agent. The virtual agent was displayed using a mixed reality headset. Our experiment involved 35 participants in a within-subject design. We show that, in the context of social interactions, mixed reality fares well against physical environments, and robots fare well against humans, barring a few technical challenges.

The integration of virtual reality (VR), augmented reality (AR), and mixed reality (MR) in urological practices and medical education has led to modern training systems that are cost-effective and with an increased expectation toward surgical performance and outcomes. VR aids the user in interacting with the virtual environment realistically by providing a three-dimensional (3D) view of the structures inside the body with high-level precision. AR enhances the real environment around users by integrating experience with virtual information over physical models and objects, which in turn has improved understanding of physiological mechanisms and anatomical structures. MR is an immersive technology that provides virtual content to interact with real elements. The field of urolithiasis has adapted the technological advancements, newer instruments, and methods to perform endourologic treatment procedures. This mini-review discusses the applications of Virtual Reality, Augmented Reality, and Mixed Reality in endourology and urolithiasis.

Advances in head tracking and stereoscopic visualization technologies have fostered the implementation of subjective display systems able to render a 3D scene perspective-corrected according to the position of the user. This enables a whole class of mixed reality applications and interaction paradigms, where the user is able to move freely around the scene and to perform tasks involving the interplay between physical and virtual objects. The accuracy and ergonomics of such tasks strongly depend on the ability of the subjective display system to offer not only a convincing 3D visual experience, but also, and mostly, an accurate rendering of the virtual scene in terms of spatial and metric relations between virtual and physical scene components. In this paper we study the role and impact of head tracking and stereo visualization in mixed reality contexts using a set of measuring tasks involving physical rulers and virtual objects, performed under different rendering conditions. Specifically, we analyze to what extent the two features contribute to give the user the correct alignment between the virtual and the real components of a 3D scene. Finally, we draw some conclusions about their impact within different scenarios.

In this paper, we investigate blended physical-digital kinesthetic feedback (or blended haptics in short) as a means for controlled three-dimensional design ideation in mixed reality (MR) environments. We define blended haptics as a spatial interaction wherein a physical object (e.g. a 3D printed human head) provides a specific design context for a user to generate ideas through physical manipulation of and on the object (e.g. drawing a digital sketch of a helmet) in the virtual environment. Using 3D wire-frame modeling as a concrete digital prototyping context, we investigate this idea of blended haptics in terms of how it supports design cognition specifically in spatial user interfaces. For this, we implemented a modeling tool as an experimental setup that allows a user to directly create curve-networks (wire-frame models) on a physical object (i.e. a contextual proxy) with one hand while simultaneously controlling the object with the other hand using a tracked turn-table. The key idea is that the user can simultaneously experience kinesthetic feedback from both the physical objects as well as the digital wire-frame models. To systematically investigate our approach, we conducted a comparative user evaluation of conceptual design tasks performed by two groups of users, one with the blended haptics (e.g. physical head and digital helmet) and the other with purely digital haptic feedback (e.g. digital head and digital helmet). Our study shows that blended haptics required less physical effort in the design task and resulted in concepts with higher novelty score as compared to using purely digital haptic feedback.

Mixed Reality (MR) is the next evolution of human interacting with the computer as MR has the ability to combine the physical environment and digital environment and making them coexist with each other. Interaction is still a huge research area in Augmented Reality (AR) but very less in MR, this is due to current advanced MR display techniques still not robust and intuitive enough to let the user to naturally interact with 3D content. New techniques on user interaction have been widely studied, the advanced technique in interaction when the system able to invoke more than one input modalities. Multimodal interaction undertakes to deliver intuitive multiple objects manipulation with gestures. This paper discusses the multimodal interaction technique using gesture and speech which the proposed experimental setup to implement multimodal in the MR interface. The real hand gesture is combined with speech inputs in MR to perform spatial object manipulations. The paper explains the implementation stage that involves interaction using gesture and speech inputs to enhance user experience in MR workspace. After acquiring gesture input and speech commands, spatial manipulation for selection and scaling using multimodal interaction has been invoked, and this paper ends with a discussion.

Despite the advances in machine perception, semantic scene understanding is still a limiting factor in mixed reality scene composition. In this paper, we present TransforMR, a video see-through mixed reality system for mobile devices that performs 3D-pose-aware object substitution to create meaningful mixed reality scenes. In real-time and for previously unseen and unprepared real-world environments, TransforMR composes mixed reality scenes so that virtual objects assume behavioral and environment-contextual properties of replaced real-world objects. This yields meaningful, coherent, and humaninterpretable scenes, not yet demonstrated by today’s augmentation techniques. TransforMR creates these experiences through our novel pose-aware object substitution method building on different 3D object pose estimators, instance segmentation, video inpainting, and pose-aware object rendering. TransforMR is designed for use in the real-world, supporting the substitution of humans and vehicles in everyday scenes, and runs on mobile devices using just their monocular RGB camera feed as input. We evaluated TransforMR with eight participants in an uncontrolled city environment employing different transformation themes. Applications of TransforMR include real-time character animation analogous to motion capturing in professional film making, however without the need for preparation of either the scene or the actor, as well as narrative-driven experiences that allow users to explore fictional parallel universes in mixed reality. We make all of our source code and assets available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> TransforMR code release: https://github.com/MohamedKari/transformr

Mixed-Reality (MR) represents a combination of real and virtual worlds. Off-the-shelf solutions already exist for quite some time which enable the creation of MR worlds where physical and digital objects co-exist and interact in real time. However, little is still known regarding methodological and technical challenges involved in the development of room scale MR worlds with the use of high-end technologies, especially when considering real-object representation. This paper reports hands-on experiences in creating a room-scale MR world, using widespread, off-the-shelf object-tracking technologies and 3D modelling techniques, which enable free user movement, accurate real-object representation in the virtual world, as well as interactivity between real and virtual objects.

Historic preservationists are increasingly focusing their attention on the use of digital technologies to advance the field. These tools equip experts with the ability to capture, process, produce, and reproduce digital data of cultural heritage for documentation, interpretation, restoration and conservation purposes. In recent years, researchers have experimented with Mixed Reality (MR) applications to preserve cultural heritage in a non-contact and reversible manner. By allowing experts to intervene without altering extant physical objects, MR opens up countless possibilities for preservation strategies once the physical range of actions have been exhausted. However, more importantly, by mixing virtual and physical worlds, MR offers new possibilities for preserving historical transformation itself. In this paper, I will argue that MR provides the best way of preserving the authenticity of architectural objects by capturing the historical transformations that they endure.

The user's visual perception in mixed reality (MR) includes the combination of the real physical environment and virtual objects rendered in an optical see-through head mounted display (OST-HMD), so the presentation of MR rendering objects will be affected by the light from the real world; thus, rendering effect deviates from the original design intent. This paper focuses on the user's perception of transparency in mixed reality and the impact of ambient illumination on the perception of transparency in mixed reality. We conducted a psychophysical experimental study and recruited 20 participants to recognize the transparency of virtual objects under two kinds of ambient illumination via Microsoft HoloLens and select the corresponding value in the slider. Our analysis shows that the transparency of virtual objects will be underestimated when the ambient illumination is high. Additionally, the experiments also found that as the magnitude of transparency increases, the perception of transparency will be underestimated. This study can provide a reference for the design of MR interface transparency coding.

Researchers, educators, and multimedia designers need to better understand how mixing physical tangible objects with virtual experiences affects learning and science identity. In this novel study, a 3D-printed tangible that is an accurate facsimile of the sort of expensive glassware that chemists use in real laboratories is tethered to a laptop with a digitized lesson. Interactive educational content is increasingly being placed online, it is important to understand the educational boundary conditions associated with passive haptics and 3D-printed manipulables. Cost-effective printed objects would be particularly welcome in rural and low Socio-Economic (SES) classrooms. A Mixed Reality (MR) experience was created that used a physical 3D-printed haptic burette to control a computer-based chemistry titration experiment. This randomized control trial study with 136 college students had two conditions: 1) low-embodied control (using keyboard arrows), and 2) high-embodied experimental (physically turning a valve/stopcock on the 3D-printed burette). Although both groups displayed similar significant gains on the declarative knowledge test, deeper analyses revealed nuanced Aptitude by Treatment Interactions (ATIs). These interactionsfavored the high-embodied experimental group that used the MR devicefor both titration-specific posttest knowledge questions and for science efficacy and science identity. Those students with higher prior science knowledge displayed higher titration knowledge scores after using the experimental 3D-printed haptic device. A multi-modal linguistic and gesture analysis revealed that during recall the experimental participants used the stopcock-turning gesture significantly more often, and their recalls created a significantly different Epistemic Network Analysis (ENA). ENA is a type of 2D projection of the recall data, stronger connections were seen in the high embodied group mainly centering on the key hand-turning gesture. Instructors and designers should consider the multi-modal and multi-dimensional nature of the user interface, and how the addition of another sensory-based learning signal (haptics) might differentially affect lower prior knowledge students. One hypothesis is that haptically manipulating novel devices during learning may create more cognitive load. For low prior knowledge students, it may be advantageous for them to begin learning content on a more ubiquitous interface (e.g., keyboard) before moving them to more novel, multi-modal MR devices/interfaces.

Mixed reality (MR) applications should now be developed for product merchandising. In the light of recent advances in apparel 3‐D design and merchandising technology we argue that MR could now be applied to the sales of relatively low‐end, high volume items, such as clothing and especially sports equipment. The disadvantage that relatively few shoppers could experience the MR application at firsthand, could be offset by the massive leveraging effect of displaying the mixed reality merchandising events within a networked multi‐channel marketing context, which could truly begin to define a shared zone within which on‐line virtual trading can merge with the richly crafted physical environment of a department store. The existence of an MR infrastructure in a store could begin to open the retailing space to on‐line virtual visitor access. Two technical sketches of mixed reality merchandising scenarios (featuring women’s garments and ski equipment) are given as examples.