Realizing mixed-reality environments with tablets for intuitive human-robot collaboration for object manipulation tasks

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Augmented Reality and Mixed Reality technologies make us feel the real world and the virtual world are mixed with each other through visual matching. However, existing mixed reality contents using these technologies have a limitation that interact only with virtual objects and cannot interact with real objects. If a dynamic real object need to be influenced by a virtual object, or vice versa, illusion of Mixed Reality can be broken by the incongruity of the interaction. If these limitations are improved, real objects, which were always static, can operate dynamically and interact with virtual objects, so becomes difficult to distinguish between real objects and virtual objects. In this paper, we propose extended interactions in a mixed reality environment that can give illusion which is difficult to distinguish between real objects and virtual objects. And to confirm this, we implement authoring environment based Hololens.

An end-use’s experience of any software is typically influenced by the interface presented by the application to the user. For Mixed Reality Environments such as Augmented and Virtual Reality, the user interface is highly visual, and a poor interface can significantly degrade the user experience. Adequate attention is required when designing or creating interfaces and user experience within Mixed Reality Environments as traditional interface design goals and specifications often need to be adjusted. Furthermore, for mixed reality environments on Mobile devices there are additional interface constraints and considerations that would considerably improve the user experience when properly addressed. This research paper discusses the evolution(s) of user interface(s) and user experience of Augmented and Virtual Reality applications on Mobile devices and contributes a framework for improving user interfaces and experience when using Mixed Reality Environments.

In collaborative activities, collaborators can use physical objects in their shared environment as spatial cues to guide each other's attention. Collaborative mixed reality environments (MREs) include both, physical and digital objects. To study how virtual objects influence collaboration and whether they are used as spatial cues, we conducted a controlled lab experiment with 16 dyads. Results of our study show that collaborators favored the digital objects as spatial cues over the physical environment and the physical objects: Collaborators used significantly less deictic gestures in favor of more disambiguous verbal references and a decreased subjective workload when virtual objects were present. This suggests adding additional virtual objects as spatial cues to MREs to improve user experience during collaborative mixed reality tasks.

The ability to interact with virtual objects using gestures would allow users to improve their experience in Mixed Reality (MR) environments, especially when they use AR headsets. Today, MR head-mounted displays like the HoloLens integrate hand gesture based interaction allowing users to take actions in MR environments. However, the proposed interactions remain limited. In this paper, we propose to combine a Leap Motion Controller (LMC) with a HoloLens in order to improve gesture interaction with virtual objects. Two main issues are presented: an interactive calibration procedure for the coupled HoloLens-LMC device and an intuitive hand-based interaction approach using LMC data in the HoloLens environment. A set of first experiments was carried out to evaluate the accuracy and the usability of the proposed approach.

In this article, we present a client/server augmented reality (AR) system for viewing complex assembly models on mobile phones. Complex model information is located on a PC which takes care of all of the heavy AR tracking and rendering computation. A camera phone is used as a client to show this information, augmented on still images as animated view. The mobile phone interface also supports correct masking of the assembly pieces as they come together, as well as a graphics only viewing mode intended for better understanding of the assembly process. In addition to describing the mobile augmented assembly system we present results from two pilot user studies evaluating elements of the user interface. INTRODUCTION In order to remain competitive, today’s industry is employing a growing number of product variations with shorter and shorter life cycles, requiring efficient ways to handle the production planning and processing. Assembly line workers face more and more complex tasks and an increasing memory load is posed by the new production needs. Experienced workers may be able to handle several product variants at the same time, but new workers require training and guidance to handle the work tasks. Related activities such as maintenance and repair also require a lot of know-how of complex products. Guidance in assembly tasks is commonly provided by using printed blueprints or other documents. However, these are mainly two dimensional images or text while the real assembly task is three dimensional, making the relationship between parts hard to understand. Computer based technologies have been developed such as electronic guides, multimedia training material and interactive documents. However, these typically require the user to turn away from the assembly task to study a computer screen separate from the workspace. In our research we explore how augmented reality (AR) technology can be used to provide more intuitive guides to assist with assembly, repair and training tasks. With AR technology, synthetic objects can be merged with the user’s view of the real world so that the user can perform the assembly task without needing to look at a separate screen. The virtual objects appear in the correct position so that AR enables the creation of realistic looking 3D animated assembly “manuals”. Other researchers have developed prototype AR systems for supporting assembly tasks. These are typically based on head mounted displays (HMDs) connected to desktop or wearable computers. However, wearable computers are relatively expensive, they have a short battery life, and taking them into a harsh industrial environment is not always possible. Head mounted displays can also be bulky, they involve safety issues, and many workers are reluctant to wear them. To overcome these disadvantages, we are interested in developing a reliable mobile phone based AR assembly system. Today’s mobile phones have fast processors, significant memory, 3D graphics support, and high resolution displays. They contain various accessories that can be used for developing mobile AR applications such as good quality cameras, connectivity by Bluetooth, WLAN and GPRS, and various SAM.700.003.1155

Outdoor rendering is an attractive topic in computer graphics. In this paper our main concern is to reveal the interaction between sky color and virtual objects in Mixed Reality environments. Although registration and tracking are two of the main issues in building effective Augmented Reality (AR) systems the creation of more realistic virtual objects indistinguishable from their real-world counterparts is our target which is currently the ultimate goal in AR. Two classes of sky color generation are employed to reveal the outdoor-element interaction. Virtual Sky Modelling (VSM) based on the Perez Model is capable of generating the sky color in a specific location, date and time. The second technique is to generate a virtual model based on the real image of the sky which is called in this paper Real Sky Modelling (RSM). Subsequently, preprocessing of the sky color bleeding is based on the radiosity technique to give the sky color effect to the virtual objects as well as the real ones. Through designing a test AR set-up and applying software and hardware the goal of a robust generation of realistic virtual objects with effect of sky color is achieved.

Intangible Cultural Heritage (ICH) represents living cultural expressions and practices, which communities recognize as part of their cultural heritage. The preservation and transmission of ICH becomes a very important project since it is the mainspring of humanity's cultural diversity. Various approaches have been proposed to disseminate living cultural expressions and practices that utilize modern technologies beyond the mere digitization of ICH content. Mobile Augmented Reality (MAR) is a promising technology along this path that enables the overlap of digital and real-world information, in an engaging and efficient manner. However, this technology is still relatively new and has not been studied extensively in terms of user experience, interaction between users and virtual objects, and learning the historical aspects of ICH. The current paper presents a mobile augmented reality installation that illustrates the stages during the leather tanning process. The installation uses a scale model of the tannery which augments animated 3D content aiming for an educational experience. The experimental AR installation focuses on transferring the cultural value of the leather tanning process, combining the architectural heritage using the scale model and the intangible cultural heritage through digital representation. An exploratory study was performed to examine the impact of the installation on the users' engagement, experience, and learning. A number of interesting conclusions emerged about the proposed solution. Out of this experience, we outline several issues regarding the usability of the proposed augmented reality installation and the overall users' experience.

Hand-tracking technologies allow us to use our own hands to interact with real and virtual objects in Augmented Reality (AR) environments. This enables us to explore the interplay between hand-visualizations and hand-object interactions. We present a user study that examines the effect of different hand visualizations (invisible, transparent, opaque) on manipulation performance when interacting with real and virtual objects. For this, we implemented video-see-through (VST) AR-based virtual building blocks and hot wire tasks with real one-to-one counterparts that require participants to use gross and fine motor hand movements. To evaluate manipulation performance, we considered three measures: task completion time, number of collisions (hot wire task), and percentage of object displacement (building block task). Additionally, we explored the sense of agency and subjective impressions (preference, ease of interaction, successful and awkwardness) evoked by the different hand-visualizations. The results show that (1) manipulation performance is significantly higher when interacting with real objects compared to virtual ones, (2) invisible hands lead to fewer errors, higher agency, higher perceived success and ease of interaction during fine manipulation tasks with real objects, and (3) having some visualization of the virtual hands (transparent or opaque) overlayed on the real hands is preferred when manipulating virtual objects even when there are no significant performance improvements. Our empirical findings about the differences when interacting with real and virtual objects can aid hand visualization choices for manipulation tasks in AR.

Augmented reality environments face significant challenges in achieving context and semantic awareness, which are relevant for enhancing user experiences. Although machine learning has been extensively applied to areas such as object and action recognition, there is a knowledge gap in the integration of these techniques to improve contextual understanding in AR environments. This study explores how machine learning enhances object and action recognition, improving context awareness in augmented reality environments. A systematic review of fifteen studies revealed advancements in deep learning, semantic web, knowledge graphs, and multimodal interfaces, which collectively improved the capabilities of augmented reality systems. The findings include innovations in aspects such as 3D hand pose estimation, semantic gaze analysis through eye tracking, and hybrid approaches that optimize real-time processing. These developments improve augmented and mixed reality environments by enabling more realistic, interactive, and responsive applications. However, persistent challenges such as latency, scalability, and environmental adaptation underscore the need for more research. These results suggest a guide for the development of highly immersive and efficient AR/MR systems, with the potential to improve user interaction and operational capabilities.

As technology advances, people increasingly interact with virtual objects in settings such as augmented reality (AR) where the virtual layer is superimposed on top of the physical world. Similarly to interactions with physical objects, users may assign virtual objects with value, experience a sense of relatedness, and develop psychological ownership over these objects. The objective of this study is to understand how AR's unique characteristics influences the emergence of meaning and ownership perceptions amongst users. We conducted a study of users' interactions with a virtual dog over a three-week period, comparing AR and fully virtual settings. Our findings show that engagement with the application is a key determinant of the relation users develop with virtual objects. However, the effect of the background layer-whether physical or virtual-dominates the development of relatedness and ownership feelings, highlighting the importance of the real physical layer in shaping users' perceptions.

Traditional rehabilitation programs for children with cerebral palsy and acquired brain injuries aim to enhance motor and cognitive abilities through repetitive exercises, which are often perceived as tedious and demotivating. Extended reality technologies, including augmented reality (AR) and virtual reality, offer more engaging methods through exergames. However, to date, no AR exergames simultaneously integrate cognitive and motor aspects within navigational tasks. Developing these exergames necessitates rigorous methodological steps, especially when using emerging technologies such as AR. The MIDE (Multidisciplinary Iterative Design of Exergames) framework advocates a participatory design approach, involving users from the outset, the objective being to optimize the interface and validate game mechanics through user experience (UX) assessment. Some researchers initially test these mechanisms on healthy children before applying them to clinical populations. This study aims to evaluate the UX of our AR exergames, consisting of two games (AR Corsi and AR Zoo), in typically developing children. Typically developing children participated in two 1.5-hour sessions. During each session, they played one of two AR games using the Microsoft HoloLens 2 headset: AR Corsi and AR Zoo, both of which are designed to engage executive functions and motor skills through navigational capabilities. UX was assessed after each session using the following measures: System Usability Scale scores for usability, AttrakDiff for attractiveness and game quality, MeCue for emotional experience, and Rating scale of Perceived Exertion for Children for pre- and postsession mental and physical fatigue. A total of 27 participants (mean age 11.9, SD 1.2 years) were included in the study. Mean System Usability Scale scores were 79.9 (SD 11.4) for AR Corsi and 76.3 (SD 12.1) for AR Zoo, indicating good usability. The AttrakDiff questionnaire yielded favorable results, with scores between 1 and 3 for overall attractiveness, pragmatic quality, and stimulation for both AR games. However, the hedonic quality "identity" received neutral scores (mean 0.6, SD 0.5 for AR Corsi and mean 0.7, SD 0.8 for AR Zoo). The MeCue emotions module yielded average scores of 5.2 (SD 0.7) for AR Corsi and 5.3 (SD 0.8) for AR Zoo, significantly exceeding the theoretical mean of 4 (P<.001). We observed a significant effect of physical fatigue (P=.02) and mental fatigue (P=.002) after exposure to both games. A comparative analysis of UX between the two games showed no significant differences. This study demonstrates that our exergame, comprising two AR games, is user-friendly and well-received by typically developing children, eliciting positive emotions and overall appeal. Although some children reported fatigue, favorable UX evaluation confirms the validity of the game's content and mechanisms, suggesting its suitability for use among children with cerebral palsy and acquired brain injuries.

The term digital twin has gained a lot of traction and has seen an increase in interest in recent years in both academia and industry. The concept of a digital twin allows a person to interact with a digital object that is created similarly to a physical object so that the interaction feels the same as interacting with a physical object. Synchronization of real-world behavior in a system known as the Cyber-Physical System. Often data is combined with a 3D Computer-Aided Design model to improve data visualization and make interactions more realistic. One of the tools that can be used in this visualization is Augmented Reality. Research on the combination of AR and digital twin is still a little done, even though the combination of the two technologies can promise interaction with virtual objects in the application of digital twin becomes more realistic so that in the process of monitoring, simulation, maintenance, and decision making can be more accurate. The main focus of this paper is how the framework or model of a combination of digital twin technologies and augmented reality can help with data and information understanding and decision making.

This study is about the new expressive techniques for promotional videos for culture contents and aims to propose a new design model for making effective promotional videos using augmented reality technology and E-museum concepts. Augmented reality is one of the up-rising intelligent information technology, but has been barely used for promotional videos. However the technology enables the interaction between physical objects in the real world and virtual image though digital devices. Augmented reality opens the door for connecting real and virtual world, and invites humans to the virtual environment. There are growing numbers attempt to move off-line culture contents into on-line space. Considering the connection between culture contents and digital environment, more studies about the convergence is necessary in the cultural-art industry, especially for marketing business. Therefore, this study aims to take a brief look at augmented reality technology, and proposes the new experimental model and expressive techniques for cultural-art organizations’ promotional video, which set the experiential learning environment, and enables consumers to take contents actively.

Location-based augmented reality (AR) games have recently become massively popular, generating billions of dollars in revenue during the past five years. These games augment geographical areas with playful content, which makes the playing experience dependent on both the game and the playing location. Existing games have employed various methods for increasing the connection between the game and the physical world, such as (1) an AR mode that lets players see virtual objects in the physical world through the lens of their mobile device; and (2) connecting the game’s virtual points of interest (PoIs) to physical world objects. To explore how these influence the playing experience, we surveyed players (N = 74) across four popular location-based AR games. We analyzed the data using a thematic clustering approach, which resulted in five central themes. Among our findings was that only 7% of the surveyed players regularly used or appreciated AR features that made use of the mobile device’s camera. The most often given reason for this was that they slow down progression. By contrast, players appreciated location-based features and the augmentation of digital content on a map interface. Connecting the virtual content to physical world objects was seen to have the benefits of (1) supporting social meetings; (2) enabling learning about the real world; and (3) increasing the meaningfulness of the virtual content.