The Future of Mixed Reality Interactions

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Mixed Reality (MR) is the next evolution of human interacting with the computer as MR has the ability to combine the physical environment and digital environment and making them coexist with each other. Interaction is still a huge research area in Augmented Reality (AR) but very less in MR, this is due to current advanced MR display techniques still not robust and intuitive enough to let the user to naturally interact with 3D content. New techniques on user interaction have been widely studied, the advanced technique in interaction when the system able to invoke more than one input modalities. Multimodal interaction undertakes to deliver intuitive multiple objects manipulation with gestures. This paper discusses the multimodal interaction technique using gesture and speech which the proposed experimental setup to implement multimodal in the MR interface. The real hand gesture is combined with speech inputs in MR to perform spatial object manipulations. The paper explains the implementation stage that involves interaction using gesture and speech inputs to enhance user experience in MR workspace. After acquiring gesture input and speech commands, spatial manipulation for selection and scaling using multimodal interaction has been invoked, and this paper ends with a discussion.

Mixed reality offers new potentials for social interaction experiences with virtual agents. In addition, it can be used to experiment with the design of physical robots. However, while previous studies have investigated comfortable social distances between humans and artificial agents in real and virtual environments, there is little data with regards to mixed reality environments. In this paper, we conducted an experiment in which participants were asked to walk up to an agent to ask a question, in order to investigate the social distances maintained, as well as the subject's experience of the interaction. We manipulated both the embodiment of the agent (robot vs. human and virtual vs. physical) as well as closed vs. open posture of the agent. The virtual agent was displayed using a mixed reality headset. Our experiment involved 35 participants in a within-subject design. We show that, in the context of social interactions, mixed reality fares well against physical environments, and robots fare well against humans, barring a few technical challenges.

The 2 nd International Workshop on Virtual, Augmented, and Mixed Reality for Human-Robot Interactions (VAM-HRI) will bring together HRI, Robotics, and Mixed Reality researchers to identify challenges in mixed reality interactions between humans and robots. Topics relevant to the workshop include development of robots that can interact with humans in mixed reality, use of virtual reality for developing interactive robots, the design of new augmented reality interfaces that mediate communication between humans and robots, comparisons of the capabilities and perceptions of robots and virtual agents, and best design practices. VAM-HRI was held for the first time at HRI 2018, where it served as the first workshop of its kind at an academic AI or Robotics conference, and served as a timely call to arms to the academic community in response to the growing promise of this emerging field. VAM-HRI 2019 will follow on the success of VAM-HRI 2018, and present new opportunities for expanding this nascent research community. Website http://vam-hri.xyz/

Blowing in the wind: Increasing social presence with a virtual human via environmental airflow interaction in mixed reality

This research offers a contribution, through art theory and practice, to an emerging culturally orientated discourse regarding mixed reality interaction. It seeks to deconstruct and analyse syncretic, hybridized agency, particularly in mixed reality data transfer systems. Recent developments in bridging autonomous relationships with digital representation, through mixed reality interfacing, have brought about the need for further analysis of these new ‘postbiological’, hybridized states of being that traverse traditional paradigms of time and space. Roy Ascott’s reconceptualisation of syncretic dialogues may facilitate further understanding of multi-layered world views, both material and metaphysical, that are emerging from our engagement with such pervasive computational technologies and post-biological systems. Syncretism has traditionally been regarded as an attempt to harmonise and analogise disparate or opposing veiwpoints (Ascott, 2010). Citing recent examples of research, this will discuss what Deleuze and Guattari have called the ‘deterritorialisation’ of the human body through its dispersion into multiple reality manifestations, in relation to how mixed reality data transfer might constitute a ‘reterritorialising’ effect on our syncretic understanding of post-biological digital identity (Delueze and Guatarri, 1980).

This study aimed to analyze the impact of design classes based on mixed reality (MR) on the learning experiences and outcomes of elementary school students and to explore the characteristics of human-technology interactions in the MR environment. The research was conducted with 24 sixth-grade students at S Elementary School in Seoul, allowing learners to experience creative learning in an immersive environment where virtual and real worlds are combined through MR technology. The results show that MR-based design education significantly enhanced students' spatial understanding and problem-solving abilities, and positively influenced collaborative learning and creative interaction. Additionally, human-technology interaction in the MR environment provided students with the experience of handling both virtual and physical objects simultaneously, contributing to the improvement of digital literacy and collaborative problem-solving skills. These findings suggest the potential of MR technology as an innovative educational method suitable for the digital age, highlighting its applicability in future educational settings.

This paper shows our work in progress towards an easy to use development environment for Mixed Reality (MR) Applications. We argue that development of MR applications is a collaboration between interaction designers who know about user requirements, and expert developers who know the intricacies of MR development. This collaboration should be supported by tools that aid both roles and ease their communication. We also argue that real MR development should allow easy migration from one hardware setup to another, since hardware greatly varies in these type of applications, and it is important to fit a solution to the particular user's requirements and context.We show the foundational concepts in our work and current Integrated Development Environment (IDE) implementation. This work is based on InTml, a domain specific language for MR applications, and Eclipse, an open source, general purpose IDE.

Anticipation Before Action: EEG-Based Implicit Intent Detection for Adaptive Gaze Interaction in Mixed Reality

Mixed reality (MR) provides exciting interaction approaches in several applications. The user experience of interacting in these visually rich environments depends highly on the way the user perceives, processes, and comprehends visual information. In this work we are investigating the differences between Field Dependent - Field Independent users towards their interaction behavior in a MR environment when they were asked to perform a specific task. A study was conducted using Microsoft HoloLens device in which participants interacted with a popular HoloLens application, modified by the authors to log user interaction data in real time. Analysis of the results demonstrates the differences in the visual processing of information, especially in visually complex environments and the impact on the user's interaction behavior.

Mixed Reality (MR) technologies provide users with an environment that incorporates virtual objects and metadata into their physical surroundings. This opens up exciting new possibilities for applications in various domains including: education, training, healthcare, and Computer Supported Cooperative Work (CSCW). Recognizing physical objects in the environment provides contextual clues that can improve user task performance and influence effective cognitive load for complex tasks. However, this facet of MR is still relatively nascent because of hardware limitations. Presently, MR headsets can only detect physical surfaces; they can not distinguish between physical artifacts that a user might need in their work environment. Machine Learning (ML) based object recognition solutions can help overcome this limitation. However, current commercially available MR devices have insufficient computational capabilities and are unable to support the state-of-the-art ML based object recognition solutions. To address these challenges, we describe a novel approach and the corresponding framework by introducing a hardware level separation between the two tasks of object recognition and rendering virtual components. Our approach leverages ML based object recognition, MR, and a portable IoT edge computing platform to support contextual awareness. We describe the implementation of this approach and provide a case study. The prototype implementation uses an edge device to process and recognize objects to get information about the user’s physical environment and provides the user with situational awareness through the MR device. Our case study uses a Christmas tree decoration activity to demonstrate the ability of the proposed framework to support assembly-type tasks. The case study establishes the applicability of the proposed approach to a variety of problem domains. Object recognition and context aware capabilities, combined with virtual indicators and instructions, provide a solution that increases usability, helps reduce human error, and improves the overall task performance.

This article explores the way audiences respond to screen-based (virtual) and embodied (robotic) entities in the mixed reality terrain of the gallery space. While it would seem that physical three-dimensional objects in a gallery space, especially self-moving objects such as robots, have a distinct advantage in the reality stakes over screen images, the author suggests that there is no hard and fast distinction between how audiences respond to robotic entities and to screen-based virtual characters. It is the ability of an artwork to respond to and ‘dialogue’ with its audience—to ‘look back’ and ‘talk back’—that is the key factor in making it an engaging and believable social partner. Artists discussed include Mari Velonaki, Stelarc, Ruairi Glynn, Karolina Sobecka and Golan Levin.

While 3D user interfaces are often designed with efficiency and accuracy in mind, artistic performances have their own very specific constraints and criteria for a successful interaction in mixed or virtual reality, which have yet to be fully understood. In this paper, we study the design of 3D interactions for a Spatial Augmented-Reality display in the context of a cyber-opera. We perform an analysis of design decisions taken during the multiple residencies and lab sessions, and we conduct a reflexive thematic analysis of interviews with the director and actors, which highlight how they integrated the story, the actors' performance, the audience experience and the technology. We identify four main criteria for designing 3D interactions in performing arts, namely their efficiency, expressiveness, contextualisation and legibility, and we derive guidelines for future research and creation.

Advances in head tracking and stereoscopic visualization technologies have fostered the implementation of subjective display systems able to render a 3D scene perspective-corrected according to the position of the user. This enables a whole class of mixed reality applications and interaction paradigms, where the user is able to move freely around the scene and to perform tasks involving the interplay between physical and virtual objects. The accuracy and ergonomics of such tasks strongly depend on the ability of the subjective display system to offer not only a convincing 3D visual experience, but also, and mostly, an accurate rendering of the virtual scene in terms of spatial and metric relations between virtual and physical scene components. In this paper we study the role and impact of head tracking and stereo visualization in mixed reality contexts using a set of measuring tasks involving physical rulers and virtual objects, performed under different rendering conditions. Specifically, we analyze to what extent the two features contribute to give the user the correct alignment between the virtual and the real components of a 3D scene. Finally, we draw some conclusions about their impact within different scenarios.

Fractions are one of the most complex and challenging notions for children and can often lead to frustration and a revulsion for mathematics in general. In this article, we present the Magic Cauldron, a Mixed Reality (MR) application, designed to help children apprehend fractions in a fun and interactive way. The proposed solution is a digital extension to a board game, called the Potion Workshop that is used to introduce fractions in more than 2000 schools in France. We put together a team, composed of the mathematics didacticians who designed the Potion Workshop, several teachers who use this game in their class, a multimedia designer and computer scientists, in order to create a MR game that would tackle several of the key notions that are still hard to grasp. In this article, we present the Design-Based method followed by this team. It offers insights on how to implicate non-computer scientists in the design of complex custom MR interactions. Through several cycles of collaborative design, involving three teachers and their students and the development of three prototypes, this method allowed us to produce a truly original MR application.