A mixed reality environment for collaborative product design and development

A novel approach in organizing product data and managing product information in collaborative product design and development environments is presented. Domains of product data and information are first discussed in the context of generic models of product development process and product design process. An approach called the Product Information Management System (PRIMAS) for organizing and managing product information is subsequently described. In the system, product data is organized in a genome-like relational database (PRIMAS-DB), comprising a matrix of two categories of factors, namely, the Information Fields and the Product Constituents. The Information Fields consist of four generic domains for organizing information for the various activities represented as Product Constituents. The operational model and architecture of PRIMAS are presented. Development of a prototype for PRIMAS and plan for further work are described.

The ability to interact with virtual objects using gestures would allow users to improve their experience in Mixed Reality (MR) environments, especially when they use AR headsets. Today, MR head-mounted displays like the HoloLens integrate hand gesture based interaction allowing users to take actions in MR environments. However, the proposed interactions remain limited. In this paper, we propose to combine a Leap Motion Controller (LMC) with a HoloLens in order to improve gesture interaction with virtual objects. Two main issues are presented: an interactive calibration procedure for the coupled HoloLens-LMC device and an intuitive hand-based interaction approach using LMC data in the HoloLens environment. A set of first experiments was carried out to evaluate the accuracy and the usability of the proposed approach.

Developing a vehicle is always a long and complex task. This is especially true for autonomous cars. Tasks performed by the driver are taken over by the vehicle and must be performed with maximum reliability. Developing these systems is a difficult task, especially due to limited testing capabilities. Testing a vehicle in a closed environment is safe and controlled, but the variety of test scenarios is limited. By using mixed reality environments, one can create a diverse environment around a real test vehicle, with traffic, obstacles, and unexpected situations. The real movement of the test vehicle allows testing decision and motion planning level vehicular functions. Information from the virtual world can be considered as input to the vehicle sensor. Mixed reality or digital twin simulation environments can greatly assist the autonomous vehicle development process and also serve as a basis for validation procedures for such systems. This article introduces a mixed reality simulation environment that integrates a real test vehicle into a virtual environment, can handle other real and virtual obstacles, contains traffic simulations, and visualizes all of these.

We visualize life-size sequential photographs of sports activities in a mixed reality (MR) environment. Wearing a video-see-through head-mounted display, an observer records the motions of a player using a handheld camera. Our system then places billboards in the three-dimensional MR space, on which the sequential photographs of the player’s motion are presented at life-size. In a user study, we found that the observers perceived the size of the motions more accurately than when viewing sequential photographs on a monitor display.

Digital hydraulics is a discrete technology that integrates advanced dynamic system controls, digital electronics, and machine learning to enhance fluid power systems’ performance, overall efficiency, and controllability. A mechanically actuated inline three-piston variable displacement digital pump was previously proposed and designed. The inline three-piston pump incorporates complex mechanical and hydraulic subsystems and highly coupled mechanisms. The complexity of the utilized subsystems poses challenges when assessing the viability of the conceptual design. Therefore, this work focuses on designing, developing, and implementing a collaborative virtual platform involving a digitized module showcasing the internal mechanical structure of the digital pump utilizing mixed reality (MR) technology. MR technology is acknowledged as the forthcoming evolution of the human–machine interface in the real–virtual environment utilizing computers and wearables. This technology permits running simulations that examine the complexity of highly coupled systems, like the digital pump, where understanding the physical phenomenon is far too intricate. The developed MR platform permits multiple users to collaborate in a synchronized immersive MR environment to study and analyze the applicability of the pump’s design and the adequacy of the operated mechanisms. The collaborative MR platform was designed and developed on the Unity game engine, employing Microsoft Azure and Photon Unity Networking to set up the synchronized MR environment. The platform involves a fully interactive virtual module on the digital pump design, developed in multiple stages using Microsoft’s Mixed Reality Tool Kit (MRTK) for Unity and deployed in the synchronized MR environment through a HoloLens 2 MR headset. A research study involving 71 participants was carried out at Purdue University. The study’s objective was to explore the impact of the collaborative MR environment on understanding the complexity and operation of the digital pump. It also sought to assess the effectiveness of MR in facilitating collaboration among fluid power stakeholders in a synchronized digital reality setting to study, diagnose, and control their complex systems. Surveys were designed and completed by all 71 participants after experiencing the MR platform. The results indicate that approximately 75% of the participants expressed positive attitudes toward their overall MR platform experience, with particular appreciation for its immersive nature and the synchronized collaborative environment it provided. More than 70% of the participants agreed that the pump’s collaborative MR platform was essential for studying and understanding the complexity and intricacy of the digital pump’s mechanical structure. Overall, the results demonstrate that the MR platform effectively facilitates the visualization of the complex pump’s internal structure, inspection of the assembly of each of the involved subsystems, and testing the applicability of the complicated mechanisms.

Collaboration and guidance are key aspects of many software tasks. In traditional desktop software, such aspects are well supported through built-in collaboration functions or general-purpose techniques such as screen and video sharing. In Mixed Reality environments, where users carry out actions in a three-dimensional space, collaboration and guidance may also be required. However, other users may or may not be using the same Mixed Reality interface. Users may not have access to the same information, the same visual representation, or the same interaction affordances. These asymmetries make communication and collaboration between users harder. To address asymmetries in Mixed Reality environments, we introduce Interactive Mixed-Dimensional Media. In these media, the visual representation of information streams can be changed between 2D and 3D. Different representations can be chosen automatically, based on context, or through associated interaction techniques that give users control over exploring spatial, temporal, and dimensional levels of detail. This ensures that any information or interaction makes sense across different dimensions, interfaces and spaces. We have deployed these techniques in three different contexts: mixed-reality telepresence for physical task instruction, video-based instruction for VR tasks, and live interaction between a VR user and a non-VR user. From these works, we show that Mixed Reality environments that provide users with interactive mixed-dimensional media interfaces improve performance and user experience in collaboration and guidance tasks.

Lower body tracking and user experience analysis are ongoing challenges in immersive technologies and biomechanics research, respectively. Particularly, in immersive technologies like mixed reality, there are limited research applications in user comfort and experience, and locomotion. In this paper, we aimed to conduct a cross-disciplinary research (by combining the elements of Biomechanics and human–computer interaction) in understanding the impact of user experience and immersive technologies induced symptoms in virtual and mixed reality environments for obstacle clearance tasks. For this, we created obstacle clearance applications for Oculus Quest and Microsoft HoloLens. Also, we conducted a user study with 12 participants under two conditions (virtual reality and mixed reality) in obstacle clearance behavior of varying heights. The results from our study indicated that the levels of immersion and enjoyment were significantly higher in virtual reality compared to mixed reality. In terms of virtual reality-induced symptoms, the participants found the mixed reality environment to be slightly more comfortable than virtual reality.

Augmented Reality and Mixed Reality technologies make us feel the real world and the virtual world are mixed with each other through visual matching. However, existing mixed reality contents using these technologies have a limitation that interact only with virtual objects and cannot interact with real objects. If a dynamic real object need to be influenced by a virtual object, or vice versa, illusion of Mixed Reality can be broken by the incongruity of the interaction. If these limitations are improved, real objects, which were always static, can operate dynamically and interact with virtual objects, so becomes difficult to distinguish between real objects and virtual objects. In this paper, we propose extended interactions in a mixed reality environment that can give illusion which is difficult to distinguish between real objects and virtual objects. And to confirm this, we implement authoring environment based Hololens.

While there is increased interest in using movement and embodiment to support learning due to the rise in theories of embodied cognition and learning, additional work needs to be done to explore how we can make sense of students collectively developing their understanding within a mixed-reality environment. In this paper, we explore embodied communication’s individual and collective functions as a way of seeing students’ learning through embodiment. We analyze data from a mixed-reality (MR) environment: Science through Technology Enhanced Play (STEP) (Danish et al., International Journal of Computer-Supported Collaborative Learning 15:49–87, 2020), using descriptive statistics and interaction analysis to explore the role of gesture and movement in student classroom activities and their pre-and post-interviews. The results reveal that students appear to develop gestures for representing challenging concepts within the classroom and then use these gestures to help clarify their understanding within the interview context. We further explore how students collectively develop these gestures in the classroom, with a focus on their communicative acts, then provide a list of individual and collective functions that are supported by student gestures and embodiment within the STEP MR environment, and discuss the functions of each act. Finally, we illustrate the value of attending to these gestures for educators and designers interested in supporting embodied learning.

In an era where non-face-to-face meetings become common, eXtended Reality(XR) is rapidly developing and filling areas that are not satisfied in online meetings based on existing photos/video method. In particular, general users are also able to easily access and use HMD-type mixed reality devices. However, the basic operations applied in HMD-type in Mixed Reality(MR) with hands as the main input tool do not have a standardized system, and each manufacturer operates in a separate response to each other's hand movements. Therefore, this study considered that a systematic hand motion matching system considering the usability and efficiency of operations performed in mixed reality was necessary, and conducted a study to clarify this. First, the basic operation performed in the MR environment and its attributes were investigated, and at the same time, the structure of the hand and the attributes of the possible hand movements were identified. Based on the identified properties, it is intended to present a system that can intuitively and efficiently match basic operation properties in the MR environment with subtle operation properties according to the structure/context of the hand.

For mixed reality applications, where reality and virtual reality are spatially merged and aligned in interactive real-time, we propose a pure voxel representation as a rendering and interaction method of choice. We show that voxels\textemdash gap-less volumetric pixels in a regular grid in space\textemdash allow for an actual user experience of a mixed reality environment, for a seamless blending of virtual and real, and for a sense of presence and co-presence in such an environment. If everything is based on voxels, even if coarse, visual coherence is achieved inherently. We argue the case for voxels by (1) conceptually defining and illustrating voxel-based mixed reality, (2) describing the computational feasibility, (3) presenting a fully functioning, low resolution prototype, (4) empirically exploring the user experience, and finally (5) discussing current work and future directions for voxel-based mixed reality. This work is not the first that utilises voxels for mixed reality, but is the first that has an holistic view on voxel-based mixed realities as an effective way of experiencing and interacting with mixed reality environments.

Mixed reality (MR) telepresence provides a shared common 3D space for networked users to enjoy real-time immersive experiences with natural interactivity and movement. Due to its high hardware and compute demands, a large-scale cloud-edge MR solution is necessary for achieving critical mass adoption, where users engage in the MR environment using low-cost cameras at the edge and complex 3D world processing is offloaded to the cloud servers. However, cloud-edge MR solutions have distinct challenges such as concurrent multiple camera support, unpredictable compute demands, and CPU/GPU contention. In this paper, we present PointPresence, an edge-compute framework for large-scale MR applications. PointPresence is deployed at an edge node, such as an O-RAN RIC or a local enterprise server, and provides low-latency MR experiences to a large number of users by incorporating intelligent camera selection for compute reduction, reactive pipeline design for adaptation to compute demand changes, and context-aware GPU sharing. Our comprehensive evaluation on an MR testbed shows that PointPresence reduces end-to-end latency by up to 3.5× and improves end user perceived visual quality by 30%.

BackgroundApplications of robotics in daily life are becoming essential by creating new possibilities in different fields, especially in the collaborative environment. The potentials of collaborative robots are tremendous as they can work in the same workspace as humans. A framework employing a top-notch technology for collaborative robots will surely be worthwhile for further research.ObjectiveThis study aims to present the development of a novel framework for the collaborative robot using mixed reality.MethodsThe framework uses Unity and Unity Hub as a cross-platform gaming engine and project management tool to design the mixed reality interface and digital twin. It also uses the Windows Mixed Reality platform to show digital materials on holographic display and the Azure mixed reality services to capture and expose digital information. Eventually, it uses a holographic device (HoloLens 2) to execute the mixed reality–based collaborative system.ResultsA thorough experiment was conducted to validate the novel framework for mixed reality–based control of a collaborative robot. This framework was successfully applied to implement a collaborative system using a 5–degree of freedom robot (xArm-5) in a mixed reality environment. The framework was stable and worked smoothly throughout the collaborative session. Due to the distributed nature of cloud applications, there is a negligible latency between giving a command and the execution of the physical collaborative robot.ConclusionsOpportunities for collaborative robots in telerehabilitation and teleoperation are vital as in any other field. The proposed framework was successfully applied in a collaborative session, and it can also be applied in other similar potential applications for robust and more promising performance.

Design is an integral component of product development, which includes manufacturing. In order to avoid costly reworks and delays, and to reduce the lead-time in product development, effective communication of design concepts among the various stakeholders such as customers, designers and engineers is of utmost importance. In this respect, functional representations provide a methodology, which can be used to describe a design or product effectively. However, function is not the only type of information that needs to be communicated among the various stakeholders during a design process. Other types of information such as design specifications and constraints, which are non-functional, also need to be taken into account. This non-functional information cannot be adequately expressed using existing functional representations. Accordingly, in this work, a novel purpose-behaviour-structure (PBS) representation scheme to capture vital design knowledge for collaborative product development is proposed. The proposed PBS representation scheme can be used by designers working in a collaborative environment for product development. A case study on the design of a vacuum cleaner is used to illustrate the functionality of the PBS representation scheme. The details of the representation scheme are presented in this paper.

Collaborative virtual environment for product design occurs between people and often involves 3D objects. Fruitful cooperation is helped by natural and intuitive way of communication and interaction, for which responsiveness and consistency are leading factors. In order to guarantee consistency and improve responsiveness, a novel concurrency control scheme is first proposed. In our scheme, non-exclusive concurrent manipulations are merged so as to undo a small quantity of manipulations rather than all the operations triggered by users who are not granted, therefore, a majority of concurrent operations are collision-free and executed immediately to maximize responsiveness. Moreover, notification and visual indicator mechanism are used to avoid unnecessary concurrency collision. The implementation of collaborative virtual environment named VRCAD is then introduced in detail, including system architecture, concurrency control strategy and communication facilities. Experiment and practice show that our concurrency control scheme is effective and efficient for collaborative virtual environment involving 3D objects.