Mixed reality interaction: audience responses to robots and virtual characters

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Mixed Reality (MR) is the next evolution of human interacting with the computer as MR has the ability to combine the physical environment and digital environment and making them coexist with each other. Interaction is still a huge research area in Augmented Reality (AR) but very less in MR, this is due to current advanced MR display techniques still not robust and intuitive enough to let the user to naturally interact with 3D content. New techniques on user interaction have been widely studied, the advanced technique in interaction when the system able to invoke more than one input modalities. Multimodal interaction undertakes to deliver intuitive multiple objects manipulation with gestures. This paper discusses the multimodal interaction technique using gesture and speech which the proposed experimental setup to implement multimodal in the MR interface. The real hand gesture is combined with speech inputs in MR to perform spatial object manipulations. The paper explains the implementation stage that involves interaction using gesture and speech inputs to enhance user experience in MR workspace. After acquiring gesture input and speech commands, spatial manipulation for selection and scaling using multimodal interaction has been invoked, and this paper ends with a discussion.

Mixed reality offers new potentials for social interaction experiences with virtual agents. In addition, it can be used to experiment with the design of physical robots. However, while previous studies have investigated comfortable social distances between humans and artificial agents in real and virtual environments, there is little data with regards to mixed reality environments. In this paper, we conducted an experiment in which participants were asked to walk up to an agent to ask a question, in order to investigate the social distances maintained, as well as the subject's experience of the interaction. We manipulated both the embodiment of the agent (robot vs. human and virtual vs. physical) as well as closed vs. open posture of the agent. The virtual agent was displayed using a mixed reality headset. Our experiment involved 35 participants in a within-subject design. We show that, in the context of social interactions, mixed reality fares well against physical environments, and robots fare well against humans, barring a few technical challenges.

The 2 nd International Workshop on Virtual, Augmented, and Mixed Reality for Human-Robot Interactions (VAM-HRI) will bring together HRI, Robotics, and Mixed Reality researchers to identify challenges in mixed reality interactions between humans and robots. Topics relevant to the workshop include development of robots that can interact with humans in mixed reality, use of virtual reality for developing interactive robots, the design of new augmented reality interfaces that mediate communication between humans and robots, comparisons of the capabilities and perceptions of robots and virtual agents, and best design practices. VAM-HRI was held for the first time at HRI 2018, where it served as the first workshop of its kind at an academic AI or Robotics conference, and served as a timely call to arms to the academic community in response to the growing promise of this emerging field. VAM-HRI 2019 will follow on the success of VAM-HRI 2018, and present new opportunities for expanding this nascent research community. Website http://vam-hri.xyz/

Advances in head tracking and stereoscopic visualization technologies have fostered the implementation of subjective display systems able to render a 3D scene perspective-corrected according to the position of the user. This enables a whole class of mixed reality applications and interaction paradigms, where the user is able to move freely around the scene and to perform tasks involving the interplay between physical and virtual objects. The accuracy and ergonomics of such tasks strongly depend on the ability of the subjective display system to offer not only a convincing 3D visual experience, but also, and mostly, an accurate rendering of the virtual scene in terms of spatial and metric relations between virtual and physical scene components. In this paper we study the role and impact of head tracking and stereo visualization in mixed reality contexts using a set of measuring tasks involving physical rulers and virtual objects, performed under different rendering conditions. Specifically, we analyze to what extent the two features contribute to give the user the correct alignment between the virtual and the real components of a 3D scene. Finally, we draw some conclusions about their impact within different scenarios.

Blowing in the wind: Increasing social presence with a virtual human via environmental airflow interaction in mixed reality

The vision of always-on Mixed Reality interfaces that can be used in a continuous fashion for an entire day, depends on solving many difficult problems including display technology, comfort, computing power, batteries, localization, tracking, and spatial understanding. However, solving all those will not bring us to a truly useful experience unless we also solve the fundamental problem of how to effectively interact in Mixed Reality (MR). I believe that the solution to the MR interaction problem requires that we combine the approaches from interaction design, perceptual science, and machine learning, to yield truly novel and effective MR input and interactions. Such interfaces will need to be adaptive to the user context, believable, and computational in nature. We are at the exciting point in the technology development curve where there are still few universally accepted standards for MR input, which leaves a ton of opportunities for both researchers and practitioners.

This research offers a contribution, through art theory and practice, to an emerging culturally orientated discourse regarding mixed reality interaction. It seeks to deconstruct and analyse syncretic, hybridized agency, particularly in mixed reality data transfer systems. Recent developments in bridging autonomous relationships with digital representation, through mixed reality interfacing, have brought about the need for further analysis of these new ‘postbiological’, hybridized states of being that traverse traditional paradigms of time and space. Roy Ascott’s reconceptualisation of syncretic dialogues may facilitate further understanding of multi-layered world views, both material and metaphysical, that are emerging from our engagement with such pervasive computational technologies and post-biological systems. Syncretism has traditionally been regarded as an attempt to harmonise and analogise disparate or opposing veiwpoints (Ascott, 2010). Citing recent examples of research, this will discuss what Deleuze and Guattari have called the ‘deterritorialisation’ of the human body through its dispersion into multiple reality manifestations, in relation to how mixed reality data transfer might constitute a ‘reterritorialising’ effect on our syncretic understanding of post-biological digital identity (Delueze and Guatarri, 1980).

Anticipation Before Action: EEG-Based Implicit Intent Detection for Adaptive Gaze Interaction in Mixed Reality

Mixed reality (MR) provides exciting interaction approaches in several applications. The user experience of interacting in these visually rich environments depends highly on the way the user perceives, processes, and comprehends visual information. In this work we are investigating the differences between Field Dependent - Field Independent users towards their interaction behavior in a MR environment when they were asked to perform a specific task. A study was conducted using Microsoft HoloLens device in which participants interacted with a popular HoloLens application, modified by the authors to log user interaction data in real time. Analysis of the results demonstrates the differences in the visual processing of information, especially in visually complex environments and the impact on the user's interaction behavior.

Against the backdrop of digital transformation in education, the integration of Mixed Reality (MR) and Artificial Intelligence (AI) has fostered a new “virtual-physical symbiotic” learning environment. However, current educational tools often fail to capture learners’ dynamic needs and the characteristics of 3D immersive scenarios, resulting in a disconnect between virtual character interactions and the teaching process. Existing approaches face various limitations: rule-based models struggle with the dynamics of three-dimensional spaces; collaborative filtering algorithms overlook spatial contextual features; and deep learning models lack joint modeling of emotion and context. These shortcomings highlight insufficient multi-source data integration and weak situational awareness. To address these challenges, this study proposes a virtual character recommendation method tailored to immersive learning environments. The model is composed of a transformation layer, a self-adversarial data generation layer, an embedding representation layer, and a virtual character prediction layer. It achieves dynamic matching between virtual characters and 3D interactive scenarios by incorporating multi-source data standardization, intelligent agent-based game simulation for scenario data generation, semantic vectorization of character features, and Long Short-Term Memory (LSTM)-attention mechanism fusion. This research marks the first application of MR spatial computation and self-adversarial learning in educational role recommendation, offering a technical framework to tackle adaptation challenges in immersive scene recommendations. The proposed approach contributes both theoretical innovations and practical guidance for the advancement of intelligent education.

This study proposes a novel hands-free interaction method using multimodal gestures such as eye gazing and head gestures and deep learning for human-robot interaction (HRI) in mixed reality (MR) environments. Since human operators hold some objects for conducting tasks, there are many constrained situations where they cannot use their hands for HRI interactions. To provide more effective and intuitive task assistance, the proposed hands-free method supports coarse-to-fine interactions. Eye gazing-based interaction is used for coarse interactions such as searching and previewing of target objects, and head gesture interactions are used for fine interactions such as selection and 3D manipulation. In addition, deep learning-based object detection is applied to estimate the initial positioning of physical objects to be manipulated by the robot. The result of object detection is then combined with 3D spatial mapping in the MR environment for supporting accurate initial object positioning. Furthermore, virtual object-based indirect manipulation is proposed to support more intuitive and efficient control of the robot, compared with traditional direct manipulation (e.g., joint-based and end effector-based manipulations). In particular, a digital twin, the synchronized virtual robot of the real robot, is used to provide a preview and simulation of the real robot to manipulate it more effectively and accurately. Two case studies were conducted to confirm the originality and advantages of the proposed hands-free HRI: (1) performance evaluation of initial object positioning and (2) comparative analysis with traditional direct robot manipulations. The deep learning-based initial positioning reduces much effort for robot manipulation using eye gazing and head gestures. The object-based indirect manipulation also supports more effective HRI than previous direct interaction methods.

Interacting with real-world objects in Mixed Reality (MR) often proves difficult when they are crowded, distant, or partially occluded, hindering straightforward selection and manipulation. We observe that these difficulties stem from performing interaction directly on physical objects, where input is tightly coupled to their physical constraints. Our key insight is to decouple interaction from these constraints by introducing proxies-abstract representations of real-world objects. We embody this concept in Reality Proxy, a system that seamlessly shifts interaction targets from physical objects to their proxies during selection. Beyond facilitating basic selection, Reality Proxy uses AI to enrich proxies with semantic attributes and hierarchical spatial relationships of their corresponding physical objects, enabling novel and previously cumbersome interactions in MR - such as skimming, attribute-based filtering, navigating nested groups, and complex multi object selections - all without requiring new gestures or menu systems. We demonstrate Reality Proxy's versatility across diverse scenarios, including office information retrieval, large-scale spatial navigation, and multi-drone control. An expert evaluation suggests the system's utility and usability, suggesting that proxy-based abstractions offer a powerful and generalizable interaction paradigm for future MR systems.

The appearance of video games in modern society introduced a number of modifications in the recreational and socialization habits of both youths and adults. In particular, various studies have associated the excessive use of this media with health and social problems, being the “classical” video game often a sedentary and solitary activity. The purpose of this work is to propose a possible way to deal with the above issues, which consists in exploiting a platform for robotic gaming based on consumer hardware that is being developed with the aim to reintroduce the physical and social dimensions in digital games. The proposed solution encompasses a floor-projected Mixed Reality (MR) environment, an autonomous toy robot and a set of tangible interfaces created using proximity beacons, which are combined in a robotic game concept named RobotQuest that is meant to show how to favor an engaging room-scale interaction between players and real/virtual game elements.

The last decade has seen a significant surge in the popularity of virtual reality in countless fields and it keeps on spreading into more areas of application. While Virtual Reality (VR) is already being used in various educational settings, such as medicine and teaching, the technology is constantly being enhanced and augmented into Mixed Reality (MR) approaches. Especially in the training of certain skills, mixed reality provides numerous new training perspectives. The specific domain of special education is no exception to that fact and potential applications of VR are increasingly being explored. The aim of this work is to develop a VR trainer tor the training of special educators, in order for them to acquire new skills in specific teaching techniques. All aspects of this work are conducted in collaboration with Reykjavik University, where previous work on the topic was conducted in 2017. The goal is to improve on the existing trainer by adding new interaction modalities, such as voice input and virtual touch. In addition to that, the state of the art is pushed even further by introducing Augmented Virtuality (AV) through physical objects in a second version of this trainer. The goal of implementing a new simulator for training special education teachers is successfully achieved, as witnessed during two user tests with students and teachers from the field of special education. In the future, this VR trainer will continue to be used at Reykjavik University, superseding the previous one. The goal of introducing AV through physical objects is achieved through a viable technical solution that still needs to be simplified in its deployment before extensive user testing.