Post-Biological Agency in Real-Time Mixed Reality Data Transfer

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Mixed Reality (MR) is the next evolution of human interacting with the computer as MR has the ability to combine the physical environment and digital environment and making them coexist with each other. Interaction is still a huge research area in Augmented Reality (AR) but very less in MR, this is due to current advanced MR display techniques still not robust and intuitive enough to let the user to naturally interact with 3D content. New techniques on user interaction have been widely studied, the advanced technique in interaction when the system able to invoke more than one input modalities. Multimodal interaction undertakes to deliver intuitive multiple objects manipulation with gestures. This paper discusses the multimodal interaction technique using gesture and speech which the proposed experimental setup to implement multimodal in the MR interface. The real hand gesture is combined with speech inputs in MR to perform spatial object manipulations. The paper explains the implementation stage that involves interaction using gesture and speech inputs to enhance user experience in MR workspace. After acquiring gesture input and speech commands, spatial manipulation for selection and scaling using multimodal interaction has been invoked, and this paper ends with a discussion.

Mixed reality offers new potentials for social interaction experiences with virtual agents. In addition, it can be used to experiment with the design of physical robots. However, while previous studies have investigated comfortable social distances between humans and artificial agents in real and virtual environments, there is little data with regards to mixed reality environments. In this paper, we conducted an experiment in which participants were asked to walk up to an agent to ask a question, in order to investigate the social distances maintained, as well as the subject's experience of the interaction. We manipulated both the embodiment of the agent (robot vs. human and virtual vs. physical) as well as closed vs. open posture of the agent. The virtual agent was displayed using a mixed reality headset. Our experiment involved 35 participants in a within-subject design. We show that, in the context of social interactions, mixed reality fares well against physical environments, and robots fare well against humans, barring a few technical challenges.

The 2 nd International Workshop on Virtual, Augmented, and Mixed Reality for Human-Robot Interactions (VAM-HRI) will bring together HRI, Robotics, and Mixed Reality researchers to identify challenges in mixed reality interactions between humans and robots. Topics relevant to the workshop include development of robots that can interact with humans in mixed reality, use of virtual reality for developing interactive robots, the design of new augmented reality interfaces that mediate communication between humans and robots, comparisons of the capabilities and perceptions of robots and virtual agents, and best design practices. VAM-HRI was held for the first time at HRI 2018, where it served as the first workshop of its kind at an academic AI or Robotics conference, and served as a timely call to arms to the academic community in response to the growing promise of this emerging field. VAM-HRI 2019 will follow on the success of VAM-HRI 2018, and present new opportunities for expanding this nascent research community. Website http://vam-hri.xyz/

Blowing in the wind: Increasing social presence with a virtual human via environmental airflow interaction in mixed reality

The vision of always-on Mixed Reality interfaces that can be used in a continuous fashion for an entire day, depends on solving many difficult problems including display technology, comfort, computing power, batteries, localization, tracking, and spatial understanding. However, solving all those will not bring us to a truly useful experience unless we also solve the fundamental problem of how to effectively interact in Mixed Reality (MR). I believe that the solution to the MR interaction problem requires that we combine the approaches from interaction design, perceptual science, and machine learning, to yield truly novel and effective MR input and interactions. Such interfaces will need to be adaptive to the user context, believable, and computational in nature. We are at the exciting point in the technology development curve where there are still few universally accepted standards for MR input, which leaves a ton of opportunities for both researchers and practitioners.

Anticipation Before Action: EEG-Based Implicit Intent Detection for Adaptive Gaze Interaction in Mixed Reality

Mixed reality (MR) provides exciting interaction approaches in several applications. The user experience of interacting in these visually rich environments depends highly on the way the user perceives, processes, and comprehends visual information. In this work we are investigating the differences between Field Dependent - Field Independent users towards their interaction behavior in a MR environment when they were asked to perform a specific task. A study was conducted using Microsoft HoloLens device in which participants interacted with a popular HoloLens application, modified by the authors to log user interaction data in real time. Analysis of the results demonstrates the differences in the visual processing of information, especially in visually complex environments and the impact on the user's interaction behavior.

This article explores the way audiences respond to screen-based (virtual) and embodied (robotic) entities in the mixed reality terrain of the gallery space. While it would seem that physical three-dimensional objects in a gallery space, especially self-moving objects such as robots, have a distinct advantage in the reality stakes over screen images, the author suggests that there is no hard and fast distinction between how audiences respond to robotic entities and to screen-based virtual characters. It is the ability of an artwork to respond to and ‘dialogue’ with its audience—to ‘look back’ and ‘talk back’—that is the key factor in making it an engaging and believable social partner. Artists discussed include Mari Velonaki, Stelarc, Ruairi Glynn, Karolina Sobecka and Golan Levin.

Advances in head tracking and stereoscopic visualization technologies have fostered the implementation of subjective display systems able to render a 3D scene perspective-corrected according to the position of the user. This enables a whole class of mixed reality applications and interaction paradigms, where the user is able to move freely around the scene and to perform tasks involving the interplay between physical and virtual objects. The accuracy and ergonomics of such tasks strongly depend on the ability of the subjective display system to offer not only a convincing 3D visual experience, but also, and mostly, an accurate rendering of the virtual scene in terms of spatial and metric relations between virtual and physical scene components. In this paper we study the role and impact of head tracking and stereo visualization in mixed reality contexts using a set of measuring tasks involving physical rulers and virtual objects, performed under different rendering conditions. Specifically, we analyze to what extent the two features contribute to give the user the correct alignment between the virtual and the real components of a 3D scene. Finally, we draw some conclusions about their impact within different scenarios.

This study proposes a novel hands-free interaction method using multimodal gestures such as eye gazing and head gestures and deep learning for human-robot interaction (HRI) in mixed reality (MR) environments. Since human operators hold some objects for conducting tasks, there are many constrained situations where they cannot use their hands for HRI interactions. To provide more effective and intuitive task assistance, the proposed hands-free method supports coarse-to-fine interactions. Eye gazing-based interaction is used for coarse interactions such as searching and previewing of target objects, and head gesture interactions are used for fine interactions such as selection and 3D manipulation. In addition, deep learning-based object detection is applied to estimate the initial positioning of physical objects to be manipulated by the robot. The result of object detection is then combined with 3D spatial mapping in the MR environment for supporting accurate initial object positioning. Furthermore, virtual object-based indirect manipulation is proposed to support more intuitive and efficient control of the robot, compared with traditional direct manipulation (e.g., joint-based and end effector-based manipulations). In particular, a digital twin, the synchronized virtual robot of the real robot, is used to provide a preview and simulation of the real robot to manipulate it more effectively and accurately. Two case studies were conducted to confirm the originality and advantages of the proposed hands-free HRI: (1) performance evaluation of initial object positioning and (2) comparative analysis with traditional direct robot manipulations. The deep learning-based initial positioning reduces much effort for robot manipulation using eye gazing and head gestures. The object-based indirect manipulation also supports more effective HRI than previous direct interaction methods.

Ensuring smart safety of workers in large construction sites calls for centralized ecosystems, supported transversely by security, privacy, and trust enablers, to facilitate data sharing and protect the growing attack surface. At the same time, human-centricity has to be a core part of the process. Mixed Reality (MR) applications allow the execution of context-aware applications with advanced interaction interfaces improving decision making, data gathering, interoperability with other services, accessibility, and real-time data sharing. We leverage the advanced features of MR interfaces and interoperability with Internet of Things (IoT) systems to contribute towards a more usable, functional, and perceptive human-centric Next Generation ecosystem in large construction plants aiming to contribute to workers' health and safety (H&S) during inspection processes. In this study, an MR enabler is designed and developed to collect data from various sources of such an ecosystem. The main objectives are to identify workers in large construction sites and their medical and training records, get real-time information regarding their stress and health levels, and alert when dangerous activity is being performed, or a worker reaches an unauthorized location. At the same time, the MR enabler allows the visualization of BIM models, their digital representations and data, as well as the submission of reports. The MR enabler, connected to other components within the Next-Generation IoT ecosystem (NGIoT), will be applied to various construction sites in Poland.

Mixed reality LVC simulation: A new approach to study pedestrian behaviour

The appearance of video games in modern society introduced a number of modifications in the recreational and socialization habits of both youths and adults. In particular, various studies have associated the excessive use of this media with health and social problems, being the “classical” video game often a sedentary and solitary activity. The purpose of this work is to propose a possible way to deal with the above issues, which consists in exploiting a platform for robotic gaming based on consumer hardware that is being developed with the aim to reintroduce the physical and social dimensions in digital games. The proposed solution encompasses a floor-projected Mixed Reality (MR) environment, an autonomous toy robot and a set of tangible interfaces created using proximity beacons, which are combined in a robotic game concept named RobotQuest that is meant to show how to favor an engaging room-scale interaction between players and real/virtual game elements.

A utilizacao da tecnologia de Realidade Mista, no campo da arte e da museologia, e cada vez mais frequente. Nas duas ultimas decadas, esse tipo de tecnologia viabilizou o desdobramento espacial e temporal, de modo consciente, indissociando o plano fisico do virtual. O projeto Talking Statues (2014), permite analisar o modo como a Realidade Mista pode ser aplicada no espaco urbano, sobretudo, na construcao de um acervo interativo, alem do territorio institucional. A acao nao esta livre dos interesses e poderes, porem, reflete um novo direcionamento das tensoes quanto ao acervo e ao museu, propriamente ditos. Este artigo, no campo da Historia e Teoria da Arte Contemporânea, discute essas novas tensoes e direcionamentos, a partir do projeto artistico e interativo, realizado em Londres. DOI: http://dx.doi.org/10.35700/ca.2015.ano2n3.p34-43.1830