Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations

Mixed-Reality (MR) represents a combination of real and virtual worlds. Off-the-shelf solutions already exist for quite some time which enable the creation of MR worlds where physical and digital objects co-exist and interact in real time. However, little is still known regarding methodological and technical challenges involved in the development of room scale MR worlds with the use of high-end technologies, especially when considering real-object representation. This paper reports hands-on experiences in creating a room-scale MR world, using widespread, off-the-shelf object-tracking technologies and 3D modelling techniques, which enable free user movement, accurate real-object representation in the virtual world, as well as interactivity between real and virtual objects.

In recent years robots have dramatically improved in functionality, but as a result their designs have also become more complicated. The increase in the complexity of the tasks and the design of the robot result in a challenging and time-consuming robot development cycle. This paper identifies requirements for improving the robot development process, focusing on facilitating a safe and reliable transition process from simulation to real world tests. We propose that, by applying the concept of Mixed Reality (MR) to robot simulation, it is possible to create a realistic, synthetic environment that enables robot software developers to experiment freely and safely. The new simulation approach, named MR simulation, offers the flexibility of creating simulation environments composed of real and virtual objects that seamlessly interact with one another. This paper contributes a generic, conceptual MR framework which is demonstrated to provide a common, standardised simulation environment for experimentation involving real and virtual objects. An MR robot simulator is implemented based on the framework, integrating standardised data interfaces in its design to increase interoperability. Evaluation results demonstrate that the MR robot simulator is reusable in three different applications. It improved simulation reliability, reduced development cost, and increased efficiency.

The last decade has seen a significant surge in the popularity of virtual reality in countless fields and it keeps on spreading into more areas of application. While Virtual Reality (VR) is already being used in various educational settings, such as medicine and teaching, the technology is constantly being enhanced and augmented into Mixed Reality (MR) approaches. Especially in the training of certain skills, mixed reality provides numerous new training perspectives. The specific domain of special education is no exception to that fact and potential applications of VR are increasingly being explored. The aim of this work is to develop a VR trainer tor the training of special educators, in order for them to acquire new skills in specific teaching techniques. All aspects of this work are conducted in collaboration with Reykjavik University, where previous work on the topic was conducted in 2017. The goal is to improve on the existing trainer by adding new interaction modalities, such as voice input and virtual touch. In addition to that, the state of the art is pushed even further by introducing Augmented Virtuality (AV) through physical objects in a second version of this trainer. The goal of implementing a new simulator for training special education teachers is successfully achieved, as witnessed during two user tests with students and teachers from the field of special education. In the future, this VR trainer will continue to be used at Reykjavik University, superseding the previous one. The goal of introducing AV through physical objects is achieved through a viable technical solution that still needs to be simplified in its deployment before extensive user testing.

Mixed Reality applications create highly immersive experiences by integrating real world objects into the virtual world and thus expanding the virtual experience by a tactile perception. Interaction with mixed reality worlds becomes more natural if the user can interact using only his hands. Hence, the question arises whether the experience in mixed reality applications can be increased if the user’s hands and body are integrated and virtually represented as well. Hence, as a proof of concept, we developed a mixed reality application in which every physical object has a respective virtual counterpart. The major concept of the development is to experience height while walking over a wooden plank and to solve an interaction task afterwards. We present an empirical study that examines the effect of a full body vs a hands vs no representation on the experience on presence and the physiological arousal. Results indicate that integrating a virtual body increases the experience of presence as well as the physiological arousal.

We present an optimization-based approach that automatically adapts Mixed Reality (MR) interfaces to different physical environments. Current MR layouts, including the position and scale of virtual interface elements, need to be manually adapted by users whenever they move between environments, and whenever they switch tasks. This process is tedious and time consuming, and arguably needs to be automated for MR systems to be beneficial for end users. We contribute an approach that formulates this challenge as a combinatorial optimization problem and automatically decides the placement of virtual interface elements in new environments. To achieve this, we exploit the semantic association between the virtual interface elements and physical objects in an environment. Our optimization furthermore considers the utility of elements for users’ current task, layout factors, and spatio-temporal consistency to previous layouts. All those factors are combined in a single linear program, which is used to adapt the layout of MR interfaces in real time. We demonstrate a set of application scenarios, showcasing the versatility and applicability of our approach. Finally, we show that compared to a naive adaptive baseline approach that does not take semantic associations into account, our approach decreased the number of manual interface adaptations by 33%.

Smart Things are commonly understood as wireless ad-hoc networked, mobile, autonomous, special purpose computing appliances, usually interacting with their environment implicitly via a variety of sensors on the input side and actuators on the output side. Such smart appliances have started to populate the with hidden or services, thus building up an of associated with real world objects. With the embedding of invisible technology into everyday things, however, also the intuitive perception of services disappears. We believe that it has potential advantages to support the perception of smart appliance via novel interactive visual experiences. For this purpose we have developed and built DigiScope, a see-through based visual real time perception system for worlds to support interactive theater experience in mixed reality spaces. A case study is presented that demonstrates the use of DigiScope to percept the services of our smart Internet appliance SmartCase. Opposed to previous work on mixed reality based augmentation of reality, the DigiScope approach allows for a multiuser, collaborative real time perceptual experience.

Augmented Reality and Mixed Reality technologies make us feel the real world and the virtual world are mixed with each other through visual matching. However, existing mixed reality contents using these technologies have a limitation that interact only with virtual objects and cannot interact with real objects. If a dynamic real object need to be influenced by a virtual object, or vice versa, illusion of Mixed Reality can be broken by the incongruity of the interaction. If these limitations are improved, real objects, which were always static, can operate dynamically and interact with virtual objects, so becomes difficult to distinguish between real objects and virtual objects. In this paper, we propose extended interactions in a mixed reality environment that can give illusion which is difficult to distinguish between real objects and virtual objects. And to confirm this, we implement authoring environment based Hololens.

Mixed Reality (MR) technologies provide users with an environment that incorporates virtual objects and metadata into their physical surroundings. This opens up exciting new possibilities for applications in various domains including: education, training, healthcare, and Computer Supported Cooperative Work (CSCW). Recognizing physical objects in the environment provides contextual clues that can improve user task performance and influence effective cognitive load for complex tasks. However, this facet of MR is still relatively nascent because of hardware limitations. Presently, MR headsets can only detect physical surfaces; they can not distinguish between physical artifacts that a user might need in their work environment. Machine Learning (ML) based object recognition solutions can help overcome this limitation. However, current commercially available MR devices have insufficient computational capabilities and are unable to support the state-of-the-art ML based object recognition solutions. To address these challenges, we describe a novel approach and the corresponding framework by introducing a hardware level separation between the two tasks of object recognition and rendering virtual components. Our approach leverages ML based object recognition, MR, and a portable IoT edge computing platform to support contextual awareness. We describe the implementation of this approach and provide a case study. The prototype implementation uses an edge device to process and recognize objects to get information about the user’s physical environment and provides the user with situational awareness through the MR device. Our case study uses a Christmas tree decoration activity to demonstrate the ability of the proposed framework to support assembly-type tasks. The case study establishes the applicability of the proposed approach to a variety of problem domains. Object recognition and context aware capabilities, combined with virtual indicators and instructions, provide a solution that increases usability, helps reduce human error, and improves the overall task performance.

This paper describes an approach to dealing with mutual occlusions between virtual and real objects on a table-top display. Display tables use stereoscopy to make virtual content appear to exist in 3 dimensions on or above a table top. The actual image, however, lies on the physical plane of the display table. Any real physical object introduced above this plane therefore obstructs our view of the display surface and disrupts the illusion of the virtual scene. The occlusions result between real objects and the display surface, not between real objects and virtual objects. For the same reason virtual objects cannot occlude real ones. Our approach uses an additional projector located near the user's head to project those parts of virtual objects that should occlude real ones directly onto the real objects. We describe possible applications and limitations of the approach and its current implementation. Despite its limitations, we believe that the proposed approach can significantly improve interaction quality and performance for mixed reality scenarios.

Advances in head tracking and stereoscopic visualization technologies have fostered the implementation of subjective display systems able to render a 3D scene perspective-corrected according to the position of the user. This enables a whole class of mixed reality applications and interaction paradigms, where the user is able to move freely around the scene and to perform tasks involving the interplay between physical and virtual objects. The accuracy and ergonomics of such tasks strongly depend on the ability of the subjective display system to offer not only a convincing 3D visual experience, but also, and mostly, an accurate rendering of the virtual scene in terms of spatial and metric relations between virtual and physical scene components. In this paper we study the role and impact of head tracking and stereo visualization in mixed reality contexts using a set of measuring tasks involving physical rulers and virtual objects, performed under different rendering conditions. Specifically, we analyze to what extent the two features contribute to give the user the correct alignment between the virtual and the real components of a 3D scene. Finally, we draw some conclusions about their impact within different scenarios.

The integration of virtual reality (VR), augmented reality (AR), and mixed reality (MR) in urological practices and medical education has led to modern training systems that are cost-effective and with an increased expectation toward surgical performance and outcomes. VR aids the user in interacting with the virtual environment realistically by providing a three-dimensional (3D) view of the structures inside the body with high-level precision. AR enhances the real environment around users by integrating experience with virtual information over physical models and objects, which in turn has improved understanding of physiological mechanisms and anatomical structures. MR is an immersive technology that provides virtual content to interact with real elements. The field of urolithiasis has adapted the technological advancements, newer instruments, and methods to perform endourologic treatment procedures. This mini-review discusses the applications of Virtual Reality, Augmented Reality, and Mixed Reality in endourology and urolithiasis.

In this paper, we investigate blended physical-digital kinesthetic feedback (or blended haptics in short) as a means for controlled three-dimensional design ideation in mixed reality (MR) environments. We define blended haptics as a spatial interaction wherein a physical object (e.g. a 3D printed human head) provides a specific design context for a user to generate ideas through physical manipulation of and on the object (e.g. drawing a digital sketch of a helmet) in the virtual environment. Using 3D wire-frame modeling as a concrete digital prototyping context, we investigate this idea of blended haptics in terms of how it supports design cognition specifically in spatial user interfaces. For this, we implemented a modeling tool as an experimental setup that allows a user to directly create curve-networks (wire-frame models) on a physical object (i.e. a contextual proxy) with one hand while simultaneously controlling the object with the other hand using a tracked turn-table. The key idea is that the user can simultaneously experience kinesthetic feedback from both the physical objects as well as the digital wire-frame models. To systematically investigate our approach, we conducted a comparative user evaluation of conceptual design tasks performed by two groups of users, one with the blended haptics (e.g. physical head and digital helmet) and the other with purely digital haptic feedback (e.g. digital head and digital helmet). Our study shows that blended haptics required less physical effort in the design task and resulted in concepts with higher novelty score as compared to using purely digital haptic feedback.

Recent technology advances in both Virtual Reality and Augmented Reality are creating an opportunity for a paradigm shift in the design of human-computer interaction systems. Delving into the Reality-Virtuality Continuum, we find Mixed Reality - systems designed to augment the physical world with virtual entities that embody characteristics of real world objects. In the medical field, Mixed Reality systems can overlay real-time and spatially accurate results onto a patient's body without the need for external screens. The complexity of these systems previously required specialized prototypes, but newly available commercial products like the Microsoft HoloLens make the technology more available. Through a combination of literature review, expert analysis, and prototyping we explore the use of Mixed Reality in healthcare. From the experience of prototyping Patiently and HoloSim, two applications for augmenting medical training, we outline considerations for the future design and development of virtual interfaces grounded in reality.

Mixed Reality is an increasingly prevalent technology that merges digital simulations with physical objects or environments. This paper presents principles for the design of mixed reality compositions. The principles are illustrated by projects and experiments by the author involving architecture and robotics.

This work explores the use of mixed-reality (MR) technology to enable situated language learning using everyday objects in the environment around the learners. The learning method is based on Presentation, Practice, and Production (PPP), which cultivates the habit of independent learning through repetition, practice, and demonstration. In our game design, the learners first interact with real-world objects via MR, and the objects’ spelling and their pronunciation will appear (Presentation), the learners repeat the pronunciation (Practice) to collect the letters of this objects, and finally the learner use the collected letters to spell out the target words (Production), which then transform into interactive 3D objects. We designed the learning experience and content tools using gestural UI, voice input, and object-to-word engine. Children in the preliminary user study found the game to be immersive, helpful in learning the spelling of the everyday objects and the target words, and additionally showed increased interests in learning about other nearby objects after playing the game.