Mixed Prototyping for Products Usability Evaluation

Determining fidelity of mixed prototypes: Effect of media and physical interaction

Augmented Reality and Mixed Reality technologies make us feel the real world and the virtual world are mixed with each other through visual matching. However, existing mixed reality contents using these technologies have a limitation that interact only with virtual objects and cannot interact with real objects. If a dynamic real object need to be influenced by a virtual object, or vice versa, illusion of Mixed Reality can be broken by the incongruity of the interaction. If these limitations are improved, real objects, which were always static, can operate dynamically and interact with virtual objects, so becomes difficult to distinguish between real objects and virtual objects. In this paper, we propose extended interactions in a mixed reality environment that can give illusion which is difficult to distinguish between real objects and virtual objects. And to confirm this, we implement authoring environment based Hololens.

We visualize life-size sequential photographs of sports activities in a mixed reality (MR) environment. Wearing a video-see-through head-mounted display, an observer records the motions of a player using a handheld camera. Our system then places billboards in the three-dimensional MR space, on which the sequential photographs of the player’s motion are presented at life-size. In a user study, we found that the observers perceived the size of the motions more accurately than when viewing sequential photographs on a monitor display.

The ability to interact with virtual objects using gestures would allow users to improve their experience in Mixed Reality (MR) environments, especially when they use AR headsets. Today, MR head-mounted displays like the HoloLens integrate hand gesture based interaction allowing users to take actions in MR environments. However, the proposed interactions remain limited. In this paper, we propose to combine a Leap Motion Controller (LMC) with a HoloLens in order to improve gesture interaction with virtual objects. Two main issues are presented: an interactive calibration procedure for the coupled HoloLens-LMC device and an intuitive hand-based interaction approach using LMC data in the HoloLens environment. A set of first experiments was carried out to evaluate the accuracy and the usability of the proposed approach.

Collaboration and guidance are key aspects of many software tasks. In traditional desktop software, such aspects are well supported through built-in collaboration functions or general-purpose techniques such as screen and video sharing. In Mixed Reality environments, where users carry out actions in a three-dimensional space, collaboration and guidance may also be required. However, other users may or may not be using the same Mixed Reality interface. Users may not have access to the same information, the same visual representation, or the same interaction affordances. These asymmetries make communication and collaboration between users harder. To address asymmetries in Mixed Reality environments, we introduce Interactive Mixed-Dimensional Media. In these media, the visual representation of information streams can be changed between 2D and 3D. Different representations can be chosen automatically, based on context, or through associated interaction techniques that give users control over exploring spatial, temporal, and dimensional levels of detail. This ensures that any information or interaction makes sense across different dimensions, interfaces and spaces. We have deployed these techniques in three different contexts: mixed-reality telepresence for physical task instruction, video-based instruction for VR tasks, and live interaction between a VR user and a non-VR user. From these works, we show that Mixed Reality environments that provide users with interactive mixed-dimensional media interfaces improve performance and user experience in collaboration and guidance tasks.

In collaborative activities, collaborators can use physical objects in their shared environment as spatial cues to guide each other's attention. Collaborative mixed reality environments (MREs) include both, physical and digital objects. To study how virtual objects influence collaboration and whether they are used as spatial cues, we conducted a controlled lab experiment with 16 dyads. Results of our study show that collaborators favored the digital objects as spatial cues over the physical environment and the physical objects: Collaborators used significantly less deictic gestures in favor of more disambiguous verbal references and a decreased subjective workload when virtual objects were present. This suggests adding additional virtual objects as spatial cues to MREs to improve user experience during collaborative mixed reality tasks.

A mixed reality environment for collaborative product design and development

Developing a vehicle is always a long and complex task. This is especially true for autonomous cars. Tasks performed by the driver are taken over by the vehicle and must be performed with maximum reliability. Developing these systems is a difficult task, especially due to limited testing capabilities. Testing a vehicle in a closed environment is safe and controlled, but the variety of test scenarios is limited. By using mixed reality environments, one can create a diverse environment around a real test vehicle, with traffic, obstacles, and unexpected situations. The real movement of the test vehicle allows testing decision and motion planning level vehicular functions. Information from the virtual world can be considered as input to the vehicle sensor. Mixed reality or digital twin simulation environments can greatly assist the autonomous vehicle development process and also serve as a basis for validation procedures for such systems. This article introduces a mixed reality simulation environment that integrates a real test vehicle into a virtual environment, can handle other real and virtual obstacles, contains traffic simulations, and visualizes all of these.

Providing natural hand interaction between a virtual object and a user on Augmented Reality is a major issue to manipulate a rendered object in a convenient way. Conventional 2D image-based recognition and interaction technique in AR has a limitation to perform a natural interaction between the user and the virtual object. In this paper, we present a stereo-vision based natural 3D hand interaction with the augmented object. In the proposed 3D hand interaction approach, 3D hand location and finger direction can be easily obtained by using stereo-vision technique while user hand is approaching to the virtual object. Two types of hand manipulation for the augmented object such as the hand pointing and hand pinching are defined. The collision detection between user hand and the virtual object is determined by using a simple ray casting emitted from the user's finger-point against the virtual object in the case of hand pointing. In the hand pinching, the collision is occurred when the thumb and the index finger are approaching to the object and the degree of angle between two fingers becomes a predetermined value. From the experiments, the proposed 3D hand interaction method can control the virtual object in a natural way rather than using a vision-based 2D hand interaction since the stereo-vision technique can obtain the depth information from the AR environments.

Outdoor rendering is an attractive topic in computer graphics. In this paper our main concern is to reveal the interaction between sky color and virtual objects in Mixed Reality environments. Although registration and tracking are two of the main issues in building effective Augmented Reality (AR) systems the creation of more realistic virtual objects indistinguishable from their real-world counterparts is our target which is currently the ultimate goal in AR. Two classes of sky color generation are employed to reveal the outdoor-element interaction. Virtual Sky Modelling (VSM) based on the Perez Model is capable of generating the sky color in a specific location, date and time. The second technique is to generate a virtual model based on the real image of the sky which is called in this paper Real Sky Modelling (RSM). Subsequently, preprocessing of the sky color bleeding is based on the radiosity technique to give the sky color effect to the virtual objects as well as the real ones. Through designing a test AR set-up and applying software and hardware the goal of a robust generation of realistic virtual objects with effect of sky color is achieved.

Lower body tracking and user experience analysis are ongoing challenges in immersive technologies and biomechanics research, respectively. Particularly, in immersive technologies like mixed reality, there are limited research applications in user comfort and experience, and locomotion. In this paper, we aimed to conduct a cross-disciplinary research (by combining the elements of Biomechanics and human–computer interaction) in understanding the impact of user experience and immersive technologies induced symptoms in virtual and mixed reality environments for obstacle clearance tasks. For this, we created obstacle clearance applications for Oculus Quest and Microsoft HoloLens. Also, we conducted a user study with 12 participants under two conditions (virtual reality and mixed reality) in obstacle clearance behavior of varying heights. The results from our study indicated that the levels of immersion and enjoyment were significantly higher in virtual reality compared to mixed reality. In terms of virtual reality-induced symptoms, the participants found the mixed reality environment to be slightly more comfortable than virtual reality.

An end-use’s experience of any software is typically influenced by the interface presented by the application to the user. For Mixed Reality Environments such as Augmented and Virtual Reality, the user interface is highly visual, and a poor interface can significantly degrade the user experience. Adequate attention is required when designing or creating interfaces and user experience within Mixed Reality Environments as traditional interface design goals and specifications often need to be adjusted. Furthermore, for mixed reality environments on Mobile devices there are additional interface constraints and considerations that would considerably improve the user experience when properly addressed. This research paper discusses the evolution(s) of user interface(s) and user experience of Augmented and Virtual Reality applications on Mobile devices and contributes a framework for improving user interfaces and experience when using Mixed Reality Environments.

This paper describes an interactive tool for evaluating assembly sequences using the novel human-computer interface of augmented reality. The goal is to be able to consider various sequencing alternatives of the manufacturing design process by manipulating both virtual and real prototype components. The augmented reality-based assembly evaluation tool would allow a manufacturing engineer to interact with the assembly planner while manipulating the real and virtual prototype components in an assembly environment. Information from the assembly planner can be displayed directly superimposed on the real. A sensing technique is proposed that uses computer vision along with a system of markers for automatically monitoring the assembly state as the user manipulates the assembly components. An implemented system called AREAS is described. The advantage of using mixed prototyping and augmented reality as means of capturing human intuition in assembly planning is also discussed.

We investigate the efficacy of incorporating real-time feedback of user performance within mixed-reality environments (MREs) for training real-world tasks with tightly coupled cognitive and psychomotor components. This paper presents an approach to providing real-time evaluation and visual feedback of learner performance in an MRE for training clinical breast examination (CBE). In a user study of experienced and novice CBE practitioners (n = 69), novices receiving real-time feedback performed equivalently or better than more experienced practitioners in the completeness and correctness of the exam. A second user study (n = 8) followed novices through repeated practice of CBE in the MRE. Results indicate that skills improvement in the MRE transfers to the real-world task of CBE of human patients. This initial case study demonstrates the efficacy of MREs incorporating real-time feedback for training real-world cognitive-psychomotor tasks.

Visual coherence between real and virtual objects is important in mixed reality (MR), and illumination consistency is one of the key aspects to achieve coherence. Apart from matching the illumination of the virtual objects with the real environments, the change of illumination on the real scenes produced by the inserted virtual objects should also be considered but is difficult to compute in real-time due to the heavy computation demands of global illumination. In this work, we propose delta path tracing (DPT), which only computes the radiance blocked by the virtual objects from the light sources at the primary hit points of Monte Carlo path tracing, then combines the blocked radiance and multi-bounce indirect illumination with the image of the real scene. Multiple importance sampling (MIS) between BRDF and environment map is performed to handle all-frequency environment maps captured by a panorama camera. Compared to conventional differential rendering methods, our method can remarkably reduce the number of times required to access the environment map and avoid rendering scenes twice. Therefore, the performance can be significantly improved. We implement our method using hardware-accelerated ray tracing on modern GPUs, and the results demonstrate that our method can render global illumination at real-time frame rates and produce plausible visual coherence between real and virtual objects in MR environments.