Blended Physical-Digital Kinesthetic Feedback for Mixed Reality-Based Conceptual Design-In-Context

Developing a vehicle is always a long and complex task. This is especially true for autonomous cars. Tasks performed by the driver are taken over by the vehicle and must be performed with maximum reliability. Developing these systems is a difficult task, especially due to limited testing capabilities. Testing a vehicle in a closed environment is safe and controlled, but the variety of test scenarios is limited. By using mixed reality environments, one can create a diverse environment around a real test vehicle, with traffic, obstacles, and unexpected situations. The real movement of the test vehicle allows testing decision and motion planning level vehicular functions. Information from the virtual world can be considered as input to the vehicle sensor. Mixed reality or digital twin simulation environments can greatly assist the autonomous vehicle development process and also serve as a basis for validation procedures for such systems. This article introduces a mixed reality simulation environment that integrates a real test vehicle into a virtual environment, can handle other real and virtual obstacles, contains traffic simulations, and visualizes all of these.

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Virtual and mixed reality technology has advanced tremendously over the past several years. This nascent medium has the potential to transform how people communicate over distance, train for unfamiliar tasks, operate in challenging environments, and how they visualize, interact, and make decisions based on complex data. At the same time, the marketplace has experienced a proliferation of network-connected devices and generalized sensors that are becoming increasingly accessible and ubiquitous. As the Internet of Things expands to encompass a predicted 50 billion connected devices by 2020, the volume and complexity of information generated in pervasive and virtualized environments will continue to grow exponentially. The convergence of these trends demands a theoretically grounded research agenda that can address emerging challenges for human-information interaction (HII). Virtual and mixed reality environments can provide controlled settings where HII phenomena can be observed and measured, new theories developed, and novel algorithms and interaction techniques evaluated. In this paper, we describe the intersection of pervasive computing with virtual and mixed reality, identify current research gaps and opportunities to advance the fundamental understanding of HII, and discuss implications for the design and development of cyber-human systems for both military and civilian use.

Presents a combined system for hand and finger kinesthetic feedback for the exploration and manipulation of different kinds of rigid objects in virtual environments. A three-degrees-of-freedom haptic interface, called DeKiFeD3 (Desktop Kinesthetic Feedback Device 3), with the capability of generating kinesthetic hand feedback, is combined with a commercial haptic glove to generate additional kinesthetic feedback to the fingers of a human operator. This system results in a new quality of kinesthetic perception and needs appropriate rendering algorithms, as described in this paper. The paper also presents an approach to improve the operator's immersion into a virtual scenario by adding auditory and visual feedback.

Lower body tracking and user experience analysis are ongoing challenges in immersive technologies and biomechanics research, respectively. Particularly, in immersive technologies like mixed reality, there are limited research applications in user comfort and experience, and locomotion. In this paper, we aimed to conduct a cross-disciplinary research (by combining the elements of Biomechanics and human–computer interaction) in understanding the impact of user experience and immersive technologies induced symptoms in virtual and mixed reality environments for obstacle clearance tasks. For this, we created obstacle clearance applications for Oculus Quest and Microsoft HoloLens. Also, we conducted a user study with 12 participants under two conditions (virtual reality and mixed reality) in obstacle clearance behavior of varying heights. The results from our study indicated that the levels of immersion and enjoyment were significantly higher in virtual reality compared to mixed reality. In terms of virtual reality-induced symptoms, the participants found the mixed reality environment to be slightly more comfortable than virtual reality.

Digital hydraulics is a discrete technology that integrates advanced dynamic system controls, digital electronics, and machine learning to enhance fluid power systems’ performance, overall efficiency, and controllability. A mechanically actuated inline three-piston variable displacement digital pump was previously proposed and designed. The inline three-piston pump incorporates complex mechanical and hydraulic subsystems and highly coupled mechanisms. The complexity of the utilized subsystems poses challenges when assessing the viability of the conceptual design. Therefore, this work focuses on designing, developing, and implementing a collaborative virtual platform involving a digitized module showcasing the internal mechanical structure of the digital pump utilizing mixed reality (MR) technology. MR technology is acknowledged as the forthcoming evolution of the human–machine interface in the real–virtual environment utilizing computers and wearables. This technology permits running simulations that examine the complexity of highly coupled systems, like the digital pump, where understanding the physical phenomenon is far too intricate. The developed MR platform permits multiple users to collaborate in a synchronized immersive MR environment to study and analyze the applicability of the pump’s design and the adequacy of the operated mechanisms. The collaborative MR platform was designed and developed on the Unity game engine, employing Microsoft Azure and Photon Unity Networking to set up the synchronized MR environment. The platform involves a fully interactive virtual module on the digital pump design, developed in multiple stages using Microsoft’s Mixed Reality Tool Kit (MRTK) for Unity and deployed in the synchronized MR environment through a HoloLens 2 MR headset. A research study involving 71 participants was carried out at Purdue University. The study’s objective was to explore the impact of the collaborative MR environment on understanding the complexity and operation of the digital pump. It also sought to assess the effectiveness of MR in facilitating collaboration among fluid power stakeholders in a synchronized digital reality setting to study, diagnose, and control their complex systems. Surveys were designed and completed by all 71 participants after experiencing the MR platform. The results indicate that approximately 75% of the participants expressed positive attitudes toward their overall MR platform experience, with particular appreciation for its immersive nature and the synchronized collaborative environment it provided. More than 70% of the participants agreed that the pump’s collaborative MR platform was essential for studying and understanding the complexity and intricacy of the digital pump’s mechanical structure. Overall, the results demonstrate that the MR platform effectively facilitates the visualization of the complex pump’s internal structure, inspection of the assembly of each of the involved subsystems, and testing the applicability of the complicated mechanisms.

This abstract is a companion piece to the ninth iteration of the “Me and My VE” interactive session. The purpose of this alternative format session is to allow human factors researchers to demonstrate how they use virtual reality, mixed reality, and virtual environments to support end-user research. The session begins with each presenter providing a brief introduction of their virtual reality, mixed reality, or virtual environment application. After this introduction, attendees engage with the presenters and their virtual environments in an interactive demonstration period. This year’s demonstrations include applications ranging from designing virtual environments for accessibility to using virtual and extended reality to support medical and aviation training, content validation, and human-computer interaction studies. The goal of the session is to allow attendees to experience how human factors professionals use virtual environments to support human factors-oriented research and to learn about the exciting work being conducted in this field.

A mixed reality environment for collaborative product design and development

The ability to interact with virtual objects using gestures would allow users to improve their experience in Mixed Reality (MR) environments, especially when they use AR headsets. Today, MR head-mounted displays like the HoloLens integrate hand gesture based interaction allowing users to take actions in MR environments. However, the proposed interactions remain limited. In this paper, we propose to combine a Leap Motion Controller (LMC) with a HoloLens in order to improve gesture interaction with virtual objects. Two main issues are presented: an interactive calibration procedure for the coupled HoloLens-LMC device and an intuitive hand-based interaction approach using LMC data in the HoloLens environment. A set of first experiments was carried out to evaluate the accuracy and the usability of the proposed approach.

Though virtual reality (VR) has been advanced to certain levels of maturity in recent years, the general public, especially the population of the blind and visually impaired (BVI), still cannot enjoy the benefit provided by VR. Current VR accessibility applications have been developed either on expensive head-mounted displays or with extra accessories and mechanisms, which are either not accessible or inconvenient for BVI individuals. In this paper, we present a mobile VR app that enables BVI users to access a virtual environment on an iPhone in order to build their skills of perception and recognition of the virtual environment and the virtual objects in the environment. The app uses the iPhone on a selfie stick to simulate a long cane in VR, and applies Augmented Reality (AR) techniques to track the iPhone’s real-time poses in an empty space of the real world, which is then synchronized to the long cane in the VR environment. Due to the use of mixed reality (the integration of VR & AR), we call it the Mixed Reality cane (MR Cane), which provides BVI users auditory and vibrotactile feedback whenever the virtual cane comes in contact with objects in VR. Thus, the MR Cane allows BVI individuals to interact with the virtual objects and identify approximate sizes and locations of the objects in the virtual environment. We performed preliminary user studies with blind-folded participants to investigate the effectiveness of the proposed mobile approach and the results indicate that the proposed MR Cane could be effective to help BVI individuals in understanding the interaction with virtual objects and exploring 3D virtual environments. The MR Cane concept can be extended to new applications of navigation, training and entertainment for BVI individuals without more significant efforts.

Interacting with real-world objects in Mixed Reality (MR) often proves difficult when they are crowded, distant, or partially occluded, hindering straightforward selection and manipulation. We observe that these difficulties stem from performing interaction directly on physical objects, where input is tightly coupled to their physical constraints. Our key insight is to decouple interaction from these constraints by introducing proxies-abstract representations of real-world objects. We embody this concept in Reality Proxy, a system that seamlessly shifts interaction targets from physical objects to their proxies during selection. Beyond facilitating basic selection, Reality Proxy uses AI to enrich proxies with semantic attributes and hierarchical spatial relationships of their corresponding physical objects, enabling novel and previously cumbersome interactions in MR - such as skimming, attribute-based filtering, navigating nested groups, and complex multi object selections - all without requiring new gestures or menu systems. We demonstrate Reality Proxy's versatility across diverse scenarios, including office information retrieval, large-scale spatial navigation, and multi-drone control. An expert evaluation suggests the system's utility and usability, suggesting that proxy-based abstractions offer a powerful and generalizable interaction paradigm for future MR systems.

We visualize life-size sequential photographs of sports activities in a mixed reality (MR) environment. Wearing a video-see-through head-mounted display, an observer records the motions of a player using a handheld camera. Our system then places billboards in the three-dimensional MR space, on which the sequential photographs of the player’s motion are presented at life-size. In a user study, we found that the observers perceived the size of the motions more accurately than when viewing sequential photographs on a monitor display.

<p>In this project-based research, we aim to identify and explore innovative methods of interaction with mixed reality environments using current technology and design principles from the field of human-computer interaction. To do so, we will review relevant literature on this topic and use the Meta Quest 2 device as a standard for its features and capabilities in our observations. Our focus is on utilizing embodied interaction and hand gesture recognition to create more immersive and intuitive experiences for users by incorporating their hands and surrounding everyday objects as a user interface to communicate with the virtual environment. The resulting minimum viable product will consist of a set of interaction designs and scenario proposals that utilize these elements. While user testing is not a part of this research, we will draw on theories and principles from the fields of human-centered design and interaction design to inform our design choices. The implications of these findings for the future of mixed reality technology and its potential applications will also be discussed.</p>

Extended reality environments will exceed virtual reality and mixed reality environments and will be understood as a new level among several of the metaverse modalities that will become technology standards on the internet of the next future. Extended reality will address as well how the user's interface will experience dramatic changes in the way users manipulate digital 3D content. Interactive objects will adopt a holographic status. The use of natural tactile interactions determines the availability of our body as an integral interface that will obtain responses in several levels of perceptions, one of which is haptics. To create the meaningful experience of our body and the possible responses obtained from a spatial interaction or interface inside this metaverse, the content and its design must achieve a fully immersive dimension. In this crucial crossroad of body and interface interactions, phenomenological experiences will be vital to understand contemporary problematics and cultural changes.

The last decade has seen a significant surge in the popularity of virtual reality in countless fields and it keeps on spreading into more areas of application. While Virtual Reality (VR) is already being used in various educational settings, such as medicine and teaching, the technology is constantly being enhanced and augmented into Mixed Reality (MR) approaches. Especially in the training of certain skills, mixed reality provides numerous new training perspectives. The specific domain of special education is no exception to that fact and potential applications of VR are increasingly being explored. The aim of this work is to develop a VR trainer tor the training of special educators, in order for them to acquire new skills in specific teaching techniques. All aspects of this work are conducted in collaboration with Reykjavik University, where previous work on the topic was conducted in 2017. The goal is to improve on the existing trainer by adding new interaction modalities, such as voice input and virtual touch. In addition to that, the state of the art is pushed even further by introducing Augmented Virtuality (AV) through physical objects in a second version of this trainer. The goal of implementing a new simulator for training special education teachers is successfully achieved, as witnessed during two user tests with students and teachers from the field of special education. In the future, this VR trainer will continue to be used at Reykjavik University, superseding the previous one. The goal of introducing AV through physical objects is achieved through a viable technical solution that still needs to be simplified in its deployment before extensive user testing.