Predicting Hand-Object Interaction for Improved Haptic Feedback in Mixed Reality

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

One of the main problems of mixed reality devices is the physically correct representation of the luminance distribution for virtual objects and their shadows in the real world. In other words, restoring the correct distribution of scene luminance is one of the key parameters that allows solving the problem of correct interaction between the virtual and real worlds. The paper proposes methods for restoring the parameters of light sources. Also, the work is aimed at the creation and formation of criteria for the quality of visual perception, allowing to evaluate the synthesized image of mixed reality and to decide how natural it is from the point of view of the observer. In this article, surveys were used to create datasets using realistic software tools. In work the neural network is trained to recognize those areas of the image that do not fit into the environment and to classify this image as a class that causes visual discomfort. As criteria for the quality of visual perception, it is proposed to use estimates of the mismatch of the parameters of shadows cast by virtual objects and the distribution of luminance over these objects in the images of scenes containing models of "real" and "virtual" objects. The level of misalignment is estimated concerning the true lighting conditions of the real world. In this work, criteria for the quality of visual perception were formed and a neural network was trained, which makes it possible to decide and analyze the quality of a synthesized mixed reality image.

This study assesses the integration of mixed reality (MR) technologies in medical first responder (MFR) training, focusing on identifying key factors influencing behavioral intention to use MR systems and practical implications for technology acceptance and enhanced realism through haptic feedback. Through a user-centered design approach, involving co-creation workshops, iterative development, and evaluations in pilot and field trials across six countries, this study evaluated technology acceptance, presence, user experience, and workload among MFRs. Both quantitative measures and qualitative feedback were collected to analyze the determinants of technology acceptance and user engagement. The MED1stMR training system, developed as a result, demonstrates that performance expectancy, effort expectancy, and social presence are significant predictors of behavioral intention to use MR training systems among MFRs. High technology acceptance and positive user experience were reported, with specific emphasis on the educational value of haptic feedback in skill training. Trainer feedback highlighted the importance of real-time performance metrics and openness to AI-driven training assistance for enhancing training outcomes. The study underscores the critical role of realistic patient interaction and the importance of aligning training challenges with users’ skills to create engaging MR training environments for MFRs. Identifying factors influencing behavioral intention offers valuable insights for the development of MR training systems, suggesting a focus on social presence and interactive capabilities to improve realism and educational value. The findings advocate for the integration of adaptive training features and further exploration of AI support in scenario optimization and performance enhancement.

Tangible physical systems are more intuitive than Intangible virtual Systems. Mixed reality systems are considered as an alternative to virtual systems, bringing advantages of tangible systems into an interaction. However, past research has mainly focussed on technical aspects of incorporating pervasive-ness and immersive-ness in the virtual systems. This paper reports on an empirical study of intuitive Interaction in a Mixed Reality game system for children and the design aspects that could facilitate intuitive Interaction in such systems. A related samples Friedman’s test showed that the Mixed Reality game system demonstrated more intuitive interactions than non‐intuitive Interactions. A linear regression analysis further established that the variation in intuitive Interaction in the Mixed Reality system could be statistically significantly explained primarily by physical affordances offered by the Mixed Reality system and to a lesser extent by the perceived affordances in the system. Design guidelines to develop intuitive Mixed Reality systems are discussed. These guidelines should allow designers to exploit the wonders of advances in technology and at the same time allow users to directly interact with the physical real world. This will allow users to access maximal physical affordances, which are primary contributors to intuitive interaction in Tangible and Mixed Reality systems. Keywords: Intuitive Interaction; Mixed Reality Systems; Tangibles; Child Computer Interaction

Pepper's ghost is a well known technique in theater productions that allows the appearance of a semi- transparent and perhaps flying ghost on the stage. This technique has also been used with digital display to create a mixed reality (MR) system, where dynamic virtual objects are presented together with the static diorama. The usual pepper's ghost setup does not allow the presentation of the virtual objects in different z location. This paper describes the development of a pepper's ghost MR system that is able to display the virtual objects in varying z locations. In this implementation, we used six layers of polymer dispersed liquid crystal (PDLC) as the projection screens. By adjusting the on/off status of the PDLC layer, we were able to present objects at different depths.

In recent years robots have dramatically improved in functionality, but as a result their designs have also become more complicated. The increase in the complexity of the tasks and the design of the robot result in a challenging and time-consuming robot development cycle. This paper identifies requirements for improving the robot development process, focusing on facilitating a safe and reliable transition process from simulation to real world tests. We propose that, by applying the concept of Mixed Reality (MR) to robot simulation, it is possible to create a realistic, synthetic environment that enables robot software developers to experiment freely and safely. The new simulation approach, named MR simulation, offers the flexibility of creating simulation environments composed of real and virtual objects that seamlessly interact with one another. This paper contributes a generic, conceptual MR framework which is demonstrated to provide a common, standardised simulation environment for experimentation involving real and virtual objects. An MR robot simulator is implemented based on the framework, integrating standardised data interfaces in its design to increase interoperability. Evaluation results demonstrate that the MR robot simulator is reusable in three different applications. It improved simulation reliability, reduced development cost, and increased efficiency.

PurposeTo evaluate the accuracy of mixed reality (MR)-guided visualization technology for spinal puncture (MRsp).MethodsMRsp involved the following three steps: 1. Lumbar spine computed tomography (CT) data were obtained to reconstruct virtual 3D images, which were imported into a HoloLens (2nd gen). 2. The patented MR system quickly recognized the spatial orientation and superimposed the virtual image over the real spine in the HoloLens. 3. The operator performed the spinal puncture with structural information provided by the virtual image. A posture fixation cushion was used to keep the subjects’ lateral decubitus position consistent. 12 subjects were recruited to verify the setup error and the registration error. The setup error was calculated using the first two CT scans and measuring the displacement of two location markers. The projection points of the upper edge of the L3 spinous process (L3↑), the lower edge of the L3 spinous process (L3↓), and the lower edge of the L4 spinous process (L4↓) in the virtual image were positioned and marked on the skin as the registration markers. A third CT scan was performed to determine the registration error by measuring the displacement between the three registration markers and the corresponding real spinous process edges.ResultsThe setup errors in the position of the cranial location marker between CT scans along the left-right (LR), anterior-posterior (AP), and superior-inferior (SI) axes of the CT bed measured 0.09 ± 0.06 cm, 0.30 ± 0.28 cm, and 0.22 ± 0.12 cm, respectively, while those of the position of the caudal location marker measured 0.08 ± 0.06 cm, 0.29 ± 0.18 cm, and 0.18 ± 0.10 cm, respectively. The registration errors between the three registration markers and the subject’s real L3↑, L3↓, and L4↓ were 0.11 ± 0.09 cm, 0.15 ± 0.13 cm, and 0.13 ± 0.10 cm, respectively, in the SI direction.ConclusionThis MR-guided visualization technology for spinal puncture can accurately and quickly superimpose the reconstructed 3D CT images over a real human spine.

Augmented Reality and Mixed Reality technologies make us feel the real world and the virtual world are mixed with each other through visual matching. However, existing mixed reality contents using these technologies have a limitation that interact only with virtual objects and cannot interact with real objects. If a dynamic real object need to be influenced by a virtual object, or vice versa, illusion of Mixed Reality can be broken by the incongruity of the interaction. If these limitations are improved, real objects, which were always static, can operate dynamically and interact with virtual objects, so becomes difficult to distinguish between real objects and virtual objects. In this paper, we propose extended interactions in a mixed reality environment that can give illusion which is difficult to distinguish between real objects and virtual objects. And to confirm this, we implement authoring environment based Hololens.

The perceptual behavior of consumers on a product displayed in the market has a vital role in analyzing the importance given to that product. Therefore, various strategies have been developed to understand this consumer behavior in the selection of products. Immersive technologies like virtual, augmented, and mixed reality are among them. With the foremost feature of immersion in the virtual world and interaction of users with virtual objects, virtual reality, and augmented reality have unlocked their potential in research and a user-friendly tool for analyzing consumer behavior. In addition to these technologies, mixed reality also has a significant role in investigating consumer behavior. Studies on immersive technologies in food applications are vast, hence this review focuses on the applications of virtual, augmented, and mixed reality in the food selection behavior of consumers. The behavioral studies are elicited to develop new products based on consumer needs, to understand the shopping behavior in supermarkets for real-time usage, and to know the influence of emotions in a selection of products. The findings suggest that virtual, augmented, and mixed reality induce immersion of the users in food selection behavioral studies. Information on the technological advancements in the tools used for bringing immersion and interaction are discussed for its futuristic applications in food. Though immersive technology gives users a realistic virtual environment experience, its application in food systems is in the budding stage. More research on human response studies would contribute to its innovative and inevitable application in the future.

This paper describes an approach to dealing with mutual occlusions between virtual and real objects on a table-top display. Display tables use stereoscopy to make virtual content appear to exist in 3 dimensions on or above a table top. The actual image, however, lies on the physical plane of the display table. Any real physical object introduced above this plane therefore obstructs our view of the display surface and disrupts the illusion of the virtual scene. The occlusions result between real objects and the display surface, not between real objects and virtual objects. For the same reason virtual objects cannot occlude real ones. Our approach uses an additional projector located near the user's head to project those parts of virtual objects that should occlude real ones directly onto the real objects. We describe possible applications and limitations of the approach and its current implementation. Despite its limitations, we believe that the proposed approach can significantly improve interaction quality and performance for mixed reality scenarios.

This paper proposes an interface especially tailored to create a design oriented realistic Mixed Reality (MR) workspace. An MR environment implies a natural mixture between virtual and real objects. We eplore the extension of real objects into the virtual world by having virtual objects attached to them. Also, in order to create a realistic environment, the problems of the occlusions between real and virtual objects are addresed. While providing more intuitive input methods than the conventional desktop modelers or Virtual Reality (VR) immersive systems, our prototype system enables the user to have an accurate perception of the shapes modeled. Thus, the user can recognize correctly the spacial location and real sizes of the models.

Recent technology advances in both Virtual Reality and Augmented Reality are creating an opportunity for a paradigm shift in the design of human-computer interaction systems. Delving into the Reality-Virtuality Continuum, we find Mixed Reality - systems designed to augment the physical world with virtual entities that embody characteristics of real world objects. In the medical field, Mixed Reality systems can overlay real-time and spatially accurate results onto a patient's body without the need for external screens. The complexity of these systems previously required specialized prototypes, but newly available commercial products like the Microsoft HoloLens make the technology more available. Through a combination of literature review, expert analysis, and prototyping we explore the use of Mixed Reality in healthcare. From the experience of prototyping Patiently and HoloSim, two applications for augmenting medical training, we outline considerations for the future design and development of virtual interfaces grounded in reality.

The recent years have witnessed an increase in the use of augmented and virtual reality systems, changing the way we interact with our environments. Such systems are commonly associated with advertising, entertainment, medicine, training and education. However, with the increasing acceptance and availability of mobile and wearable devices (e.g. head-mounted displays (HMD)), the use of these technologies is moving towards professional and industrial environments, where they would be able to support employees in their daily tasks, increasing customer satisfaction and reducing business costs. This paper presents an innovative Mixed Reality (MR) system to assist field workforce in remote locations. As part of the overall implementation, the MR system uses fuzzy logic mechanisms to improve accuracy in user tracking and object monitoring, allowing the correct representation of users and objects in the Graphical User Interfaces (GUIs), and improving the experience for users.

This chapter introduces the field of Mixed Reality that merges the real and virtual worlds seamlessly. To develop a Mixed Reality system, two fundamental problems should be addressed: tracking and user interface. Tracking in Mixed Reality is defined as the task of estimating the user's viewing pose. The goal of tracking is to achieve an accurate registration between real and virtual objects. In order to allow the user to interact e±ciently with the Mixed Reality scene, interaction techniques should be designed. A wide range of Mixed Reality applications, such as medical and entertainment applications, have been identified. We will focus on the new trends and applications in the area of Mixed Reality.

Real Time Distributed Content Rendering Technique based on Agent-mediator Communication Framework for Multi-display Systems