Real Time Distributed Content Rendering Technique based on Agent-mediator Communication Framework for Multi-display Systems

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

In this study, we developed a new haptic-mixed reality intravenous (HMR-IV) needle insertion simulation system, providing a bimanual haptic interface integrated into a mixed reality system with programmable variabilities considering real clinical environments. The system was designed for nursing students or healthcare professionals to practice IV needle insertion into a virtual arm with unlimited attempts under various changing insertion conditions (e.g., skin: color, texture, stiffness, friction; vein: size, shape, location depth, stiffness, friction). To achieve accurate hand-eye coordination under dynamic mixed reality scenarios, two different haptic devices (Dexmo and Geomagic Touch) and a standalone mixed reality system (HoloLens 2) were integrated and synchronized through multistep calibration for different coordinate systems (real world, virtual world, mixed reality world, haptic interface world, HoloLens camera). In addition, force-profile-based haptic rendering proposed in this study was able to successfully mimic the real tactile feeling of IV needle insertion. Further, a global hand-tracking method, combining two depth sensors (HoloLens and Leap Motion), was developed to accurately track a haptic glove and simulate grasping a virtual hand with force feedback. We conducted an evaluation study with 20 participants (9 experts and 11 novices) to measure the usability of the HMR-IV simulation system with user performance under various insertion conditions. The quantitative results from our own metric and qualitative results from the NASA Task Load Index demonstrate the usability of our system.

Mixed reality technology has provided more possibilities for users to interact with both the virtual and real worlds. By integrating virtual elements with real scenes, mixed reality offers a novel immersive experience. Mixed reality also introduces complex perceptual and cognitive challenges when processing conflicting information from the virtual and real worlds. This study investigates the impact of color mismatches on response time in a mixed reality scenario. We recruited 15 participants to identify the font color of words while ignoring their meaning. The experiment included three levels of perceptual-conceptual matching. We performed a global comparison across all conditions, a within-group comparison, and a between-group comparison. Results show that in the mixed reality scenario, the processing of visual information relies on low-level physical properties (e.g., brightness and hue) and is modulated by high-level cognitive factors (e.g., conflict processing and expectation). These findings indicate that visual mismatches in mixed reality can significantly affect response time, particularly when perceptual salience is low and different from cognitive expectations. The results have implications for the design of mixed reality interfaces in complex environments where users must process information from the virtual and real worlds, such as head-up displays, warehouse management systems, and industrial instructional overlays.

One of the main problems of mixed reality devices is the physically correct representation of the luminance distribution for virtual objects and their shadows in the real world. In other words, restoring the correct distribution of scene luminance is one of the key parameters that allows solving the problem of correct interaction between the virtual and real worlds. The paper proposes methods for restoring the parameters of light sources. Also, the work is aimed at the creation and formation of criteria for the quality of visual perception, allowing to evaluate the synthesized image of mixed reality and to decide how natural it is from the point of view of the observer. In this article, surveys were used to create datasets using realistic software tools. In work the neural network is trained to recognize those areas of the image that do not fit into the environment and to classify this image as a class that causes visual discomfort. As criteria for the quality of visual perception, it is proposed to use estimates of the mismatch of the parameters of shadows cast by virtual objects and the distribution of luminance over these objects in the images of scenes containing models of "real" and "virtual" objects. The level of misalignment is estimated concerning the true lighting conditions of the real world. In this work, criteria for the quality of visual perception were formed and a neural network was trained, which makes it possible to decide and analyze the quality of a synthesized mixed reality image.

In this work authors investigated the vergence-accommodation conflict that may appear in mixed reality systems. There exist two main types of the mixed reality systems, one is optical see-through mixed reality systems, other one is video see-through mixed reality systems. One of significant differences between them is how the device user observes the surrounding real world. In case of the optical see-through device, it is observed directly with natural observation conditions, however in case of the video see-through systems the real world is image is observed as an image displayed on the device screen. As a result, when observing the real world the video see-though systems might be affected by the vergence-accommodation conflict. Most of existing video see-through mixed reality systems have two fixed cameras located at the front side of the device to capture the image to be displayed for each of the user eyes. However, in this case the vergence conflict appears due to the disagreement between the fixed direction of the external cameras of the mixed reality device with the constantly changing direction of the human eyes. In the current work authors propose the special image processing methods to correct this disagreement and recover the image with correct vergence basing on the RGB images from cameras and real-world depth estimation. Authors considered different number and placement of the front cameras including the solutions using single RGBD camera and multiple RGB cameras. The simulation results are presented. The pros and cons of proposed recovering methods are shown.

This study aimed to analyze the impact of design classes based on mixed reality (MR) on the learning experiences and outcomes of elementary school students and to explore the characteristics of human-technology interactions in the MR environment. The research was conducted with 24 sixth-grade students at S Elementary School in Seoul, allowing learners to experience creative learning in an immersive environment where virtual and real worlds are combined through MR technology. The results show that MR-based design education significantly enhanced students' spatial understanding and problem-solving abilities, and positively influenced collaborative learning and creative interaction. Additionally, human-technology interaction in the MR environment provided students with the experience of handling both virtual and physical objects simultaneously, contributing to the improvement of digital literacy and collaborative problem-solving skills. These findings suggest the potential of MR technology as an innovative educational method suitable for the digital age, highlighting its applicability in future educational settings.

Tangible physical systems are more intuitive than Intangible virtual Systems. Mixed reality systems are considered as an alternative to virtual systems, bringing advantages of tangible systems into an interaction. However, past research has mainly focussed on technical aspects of incorporating pervasive-ness and immersive-ness in the virtual systems. This paper reports on an empirical study of intuitive Interaction in a Mixed Reality game system for children and the design aspects that could facilitate intuitive Interaction in such systems. A related samples Friedman’s test showed that the Mixed Reality game system demonstrated more intuitive interactions than non‐intuitive Interactions. A linear regression analysis further established that the variation in intuitive Interaction in the Mixed Reality system could be statistically significantly explained primarily by physical affordances offered by the Mixed Reality system and to a lesser extent by the perceived affordances in the system. Design guidelines to develop intuitive Mixed Reality systems are discussed. These guidelines should allow designers to exploit the wonders of advances in technology and at the same time allow users to directly interact with the physical real world. This will allow users to access maximal physical affordances, which are primary contributors to intuitive interaction in Tangible and Mixed Reality systems. Keywords: Intuitive Interaction; Mixed Reality Systems; Tangibles; Child Computer Interaction

The paper is devoted to the problem of correct vergence in the mixed reality systems. There are two main types of mixed reality devices, these are optical see-through and video see-through mixed reality systems. They differ by the way how the device user observes the real-world environment: in the optical see-through mixed reality systems the real world is observed directly and as result the vergence and accommodation conflicts are absent while in the video see-through mixed reality systems there present both vergence and accommodation conflicts. The vergence conflict is caused by the fixed orientation of the device cameras that transmit the image of the real world, and the accommodation conflict is caused by the constancy of the focal length of optical systems that form the images of the real world on the human eye retina from the CCD matrices. This paper presents the research of the possibility to restore the correct convergence of a real-world image formed by video see-through mixed reality systems. The research shows how convergence can be correctly reconstructed using images captured by fixed cameras with parallel axes located at the front side of the mixed reality headset that form images for the left and right eyes. The resulting reconstructed images are almost free from any visual artifacts. Additionally, authors show how errors in scene depth map reconstruction affects the vergence restoration quality.

In an initial study, we characterized the properties of human reach motion in the presence or absence of visuo-haptic feedback in real and Immersive Virtual Environments (IVEs) or virtual reality within a participant’s maximum arm reach. Our goal is to understand how physical reaching actions to the perceived location of targets in the presence or absence of visuo-haptic feedback are different between real and virtual viewing conditions. Typically, participants reach to the perceived location of objects in the three-dimensional (3D) environment to perform selection and manipulation actions during 3D interaction in applications such as virtual assembly or rehabilitation. In these tasks, participants typically have distorted perceptual information in the IVE as compared to the real world, in part due to technological limitations such as minimal visual field of view, resolution, latency, and jitter. In an empirical evaluation, we asked the following questions: (i) how do the perceptual differences between virtual and real world affect our ability to accurately reach to the locations of 3D objects, and (ii) how do the motor responses of participants differ between the presence or absence of visual and haptic feedback? We examined factors such as velocity and distance of physical reaching behavior between the real world and IVE, both in the presence or absence of visuo-haptic information. The results suggest that physical reach responses vary systematically between real and virtual environments, especially in situations involving the presence or absence of visuo-haptic feedback. The implications of our study provide a methodological framework for the analysis of reaching motions for selection and manipulation with novel 3D interaction metaphors and to successfully characterize visuo-haptic versus non-visuo-haptic physical reaches in virtual and real-world situations.

3D multi-user virtual environments (MUVE) bring new opportunities to foreign language teaching and learning as a platform for teaching practices. Due to immersive and interactive nature of 3D MUVE, these tools have gained popularity in the field of foreign language learning and teaching. The power of virtual learning environments lies in creating immersive 3-D spaces that give users a sense of learning by doing. Moreover, the environment is interactive and is much the same as real world interaction. MUVE provides strong support for synchronous collaborations, interactions, and immersive environments for experiential and constructivist learning settings. Some key features of the virtual worlds comprise the ability to construct 3D environments that simulate real world situations, as well as the ability to have a virtual identity to create a cross-cultural self and the ability to have participants from around the world acting virtually in designated roles that convey new types of learning within this environment.

Information management is an inseparable part of the command process. The result is that the person making decisions at the command post interacts with data providing devices in various ways. Tools virtualization process can introduce a number of significant modifications in the design of solutions for management and command. The general idea involves replacing physical devices user interface with their digital representation (so-called Virtual instruments). A more advanced level of the systems “digitalization” is to use the mixed reality environments. In solutions using Augmented reality (AR) customized HMI is displayed to the operator when he approaches to each device. Identification of device is done by image recognition of photo codes. Visualization is achieved by (optical) see-through head mounted display (HMD). Control can be done for example by means of a handheld touch panel. Using the immersive virtual environment, the command center can be digitally reconstructed. Workstation requires only VR system (HMD) and access to information network. Operator can interact with devices in such a way as it would perform in real world (for example with the virtual hands). Because of their procedures (an analysis of central vision, eye tracking) MR systems offers another useful feature of reducing requirements for system data throughput. Due to the fact that at the moment we focus on the single device. Experiments carried out using Moverio BT-200 and SteamVR systems and the results of experimental application testing clearly indicate the ability to create a fully functional information system with the use of mixed reality technology.

Recent technology advances in both Virtual Reality and Augmented Reality are creating an opportunity for a paradigm shift in the design of human-computer interaction systems. Delving into the Reality-Virtuality Continuum, we find Mixed Reality - systems designed to augment the physical world with virtual entities that embody characteristics of real world objects. In the medical field, Mixed Reality systems can overlay real-time and spatially accurate results onto a patient's body without the need for external screens. The complexity of these systems previously required specialized prototypes, but newly available commercial products like the Microsoft HoloLens make the technology more available. Through a combination of literature review, expert analysis, and prototyping we explore the use of Mixed Reality in healthcare. From the experience of prototyping Patiently and HoloSim, two applications for augmenting medical training, we outline considerations for the future design and development of virtual interfaces grounded in reality.

For efficient collaboration between participants, eye gaze is seen as being critical for interaction. Teleconferencing systems such as the AcessGrid allow users to meet across geographically disparate rooms but as of now there seems no substitute for face to face meetings. This paper gives an overview of some preliminary work that looks towards integrating eye gaze into an immersive Collaborative Virtual Environment and assessing the impact that this would have on interaction between the users of such a system. An experiment was conducted to assess the difference between users abilities to judge what objects an avatar is looking at with only head gaze being viewed and also with eye and head gaze data being displayed. The results from the experiment show that eye gaze is of vital importance to the subjects correctly identifying what a person is looking at in an immersive virtual environment. This is followed by a description of how the eye tracking system has been integrated into an immersive collaborative virtual environment and some preliminary results from the use of such a system.

We demonstrate a new technique to minimize photometric differences between the color space of a display, sensor, and the real world, for mixed-reality (MR) systems where the real world is visible near or around a display. In this approach, we use an uncalibrated sensor to capture images of the display showing a series of calibration patterns. Using these images, we precompute a polynomial mapping function describing the continuous color-space transform between the real-world, sensor, and display. This mapping function can be evaluated in real-time to produce images that, when rendered on the embedded display, greatly reduce the color differences between displayed images of a mixed-reality world and the surrounding real-world environment. The color distribution of the resulting displayed images is significantly closer to that of the ground truth real-world scene as perceived through the human eye or other sensors. We demonstrate the impact of this technique when applied to a simple dynamic-geometry mixed-reality application.

Mixed Reality (MR) is an advanced technology designed to enhance users’ experience and perception of interactions with objects in both virtual and real worlds. MR has enormous potential for improving human-cyber physical systems (HCPS) in the manufacturing industry. The transdisciplinary engineering (TE) characteristics of this research emphasize integrating advanced patent mining techniques to trend analysis of innovative MR technologies and applications in the manufacturing. Despite the growing popularity of MR applications, trend analyses in this emerging technology have relied on qualitative literature reviews and insufficient patent macro-analytics. Therefore, mapping patents related to MR inventions provides technology experts with better insights of technology development in the manufacturing. This research consists of a thorough review of non-patent literature (NPL) for the construction of MR domain ontology and a systematic patent mining analysis, including patent map analysis, patent clustering, patent topic modeling, and trend analysis of MR applications. A total of 709 patents from the InnovationQ Plus patent database are systematically identified and analyzed for trend discovery. The key terminologies, top ten patent assignees, top five applicants’ country origins, top six Cooperative Patent Classifications (CPC), and seven technology clusters and topics are identified. These findings provide reliable MR-enabled HCPS suggestions for the manufacturing.