Collaborative Augmented Digital Twin: A Novel Open-Source Augmented Reality Solution for Training and Maintenance Processes in the Shipyard of the Future

Reality & Augmented Reality in Industry collects the proceedings of the 2nd Sino-German Workshop on the same topic held in Shanghai on April 16-17, 2009. The papers focus on the latest Virtual Reality (VR) / Augmented Reality (AR) technology and its application in industrial processes and presents readers with innovative methods, typical case studies and the latest information on VR/AR basic research results and industrial applications, such as 3D rendering, innovative human-machine design, VR/AR methodology and new tools for assisting in industry, virtual assembly, virtual factory, training and education, etc. The book is intended for computer scientists, IT engineers as well as researchers in Mechanical Engineering. Dr. Dengzhe Ma and Dr. Xiumin Fan are both professors at Shanghai Jiao Tong University, China; Dr.-Ing. Jrgen Gausemeier is a professor of Computer-Integrated Manufacturing at the Heinz Nixdorf Institute, University of Paderborn, Germany; Dipl.-Ing. Michael Grafe is a senior engineer in the Product Engineering Research Group at the Heinz Nixdorf Institute, University of Paderborn.

This study presents the development and evaluation of ARTechGuide, an Augmented Reality (AR)-based application designed for technical training of operators in an automotive industrial environment. The research was conducted using the Design Science Research (DSR) methodology, with application and validation in a real factory setting. The system uses mobile devices and visual recognition to provide audiovisual instructions in five critical operational scenarios. Compared to traditional methods, a 57 % reduction in setup time, a 25 % increase in execution accuracy, and a 75 % decrease in requests for help were observed. The system achieved an average of 9.25 on the User Experience (UX) scale and 4.5 on nine technical indicators evaluated by experts. Unlike commercial AR platforms, the solution adopts a local architecture, independent of constant connectivity, facilitating its adoption in environments with operational constraints. These benefits are aligned with the Industry 5.0 proposal, which values customization according to human needs.

The quest for finding the killer application for Industrial Augmented Reality(IAR) is still active. The existing solutions are ingenious but most can not be directly integrated with the existing industrial workflows. Generally, IAR applications require modifications in the industrial workflows depending on the tracking methodology. These modifications end up being an overhead for the users and deter them from using AR solutions. In this poster we propose a resourceful solution to achieve end-to-end workflow integration with minimum effort from the user end. The solution is suited for laser guided Spatial Augmented Reality(SAR) systems mainly preferred for industrial manufacturing applications. We also introduce a new concept for pose computation, which is inspired from existing mechanical concept of part alignment. The accuracy of our method is comparable to the classical marker based methods. The complete process of pose computation, from initialisation to refinement, is designed to be plug and play.

Supporting maintenance with 3D object enhanced instruction is one of the key applications of Augmented Reality (AR) in industry. For the breakthrough of AR in maintenance, it is important that the technicians themselves can create AR-instructions and perform the challenging task of placing 3D objects as they know best how to perform a task and what necessary information needs to be displayed. For this challenge, a 3D-content editor is being presented wherein a first step the 3D objects can roughly be placed using a 2D image of the machine, therefore, limiting the time required to access the machine. In a second step, the positions of the 3D objects can be fine-tuned at the machine site using live footage. The key challenges were to develop an easily accessible UI that requires no prior knowledge of AR content creation in a tool that works both with live footage and images and is usable with a touch screen and keyboard/mouse. The 3D-content editor was qualitatively assessed by technicians revealing its general applicability, but also the requirement for a lot of time to gain the necessary experience for positioning 3D objects.

Augmented Reality is considered one of the most promising technologies for Industry 4.0. Augmented Reality allows to display the needed information at the right time and to locate it in the desired space, superimposing it to the real world. In this way, it could simplify the work of industrial operators in a variety of tasks, from planning and commissioning to assembly and maintenance. Despite the growing research interest and efforts for the development of Augmented Reality applications and technologies for the industrial domain, Augmented Reality is still rarely used in real industrial procedures, often remaining at a conceptual level. In fact, developers fail to completely answer the actual needs of industries and workers because of a lack of knowledge of the context and the absence of guidelines to drive the design and development of Augmented Reality applications. One of the open issues is the choice of the proper visualization methods to display technical information in Augmented Reality, which is not trivial. Both the paper-based documentation, based on text and illustrations, and digital documentation, including CAD models and image or video-based tutorials, present different criticalities when adapted to the Augmented Reality technology. Other visual material has been developed specifically to be used in Augmented Reality, but it is difficult to use because of a lack of standards which are instead present for the former paper-based and digital documentation. In this work, we categorize and compare different Augmented Reality visual assets, illustrating their advantages and disadvantages and providing directions for future research.

Augmented Reality systems represent technological solutions allowing integration of virtual and real worlds. As the importance of information in technology rises, AR has become a great contribution to many applications. Already since the origins of AR, industry has always belonged among the key application areas. The recent advances in both wearable and portable devices supporting AR significantly increase the applicability of AR. In connection with the new challenges arising with the development of Industry 4.0, AR systems present a great tool to productivity improvement and user experience enhancement. This is a reason, why Industrial augmented reality is considered to be one of the key parts of Industry 4.0 concepts. The principal aim of this study is to describe AR technology with emphasis put on manufacturing industry and its role in Industry 4.0 initiatives.

Industrial Augmented Reality (iAR) has demonstrated its advantages to communicate technical information in the fields of maintenance, assembly, and training. However, literature is scattered among different visual assets (i.e., AR visual user interface elements associated with a real scene). In this work, we present a systematic literature review of visual assets used in these industrial fields. We searched five databases, initially finding 1757 papers. Then, we selected 122 iAR papers from 1997 to 2019 and extracted 348 visual assets. We propose a classification for visual assets according to (i) what is displayed, (ii) how it conveys information (frame of reference, color coding, animation), and, (iii) why it is used. Our review shows that product models, text and auxiliary models are, in order, the most common, with each most often used to support operating, checking and locating tasks respectively. Other visual assets are scarcely used. Product and auxiliary models are commonly rendered world-fixed, color coding is not used as often as expected, while animations are limited to product and auxiliary model. This survey provides a snapshot of over 20 years of literature in iAR, useful to understand established practices to orientate in iAR interface design and to present future research directions.

Using Augmented Reality (AR) in the production of home appliances is an understudied field, even more its use in circular economy. Further, also the application of Artificial Intelligence (AI) for validating that an AR instructed task was actually carried out correctly is an underexplored area. In this work we present the novel concept for an AR disassembly station to recover parts from returned washing machines in a circular economy approach, validating that the disassembly is carried out correctly using a YOLO-based AI model. A last challenge, tackled in this work resulting from the circular economy background, is the adaptability of the AR instructions that need to change depending on the varying conditions of the recovered parts in units of the same model.

In the recent years, many industrial augmented reality (IAR) applications are shifting from video to still images to create a mixed view. This new type of application is called photo-based augmented reality. In order to guarantee the success of these applications, a simple and efficient registration method is required. We present a new method to register an image to a CAD model using a single keyframe. This registration is based on sparse 3D information from the model linked to the keyframe during its offline registration. We demonstrate this method in our in-house IAR software for visual inspection and documentation: VID.

Cyber-physical systems and automation play a major role in Industry 4.0. However, due to its flexibility and high responsiveness to short-term changes in the assembly of complex and varied products, manual work cannot be neglected. Cognitive assistance systems relieve the mental stress of a worker and thus help prevent mistakes during the assembly process. Systems utilizing augmented reality (AR) technologies are suitable for manual operations. In order to motivate the development needs of industrial augmented reality (IAR) applications, a prototype is described below. This paper focuses on the conceptual development and the implementation of a prototype assistance system. By using AR glasses and a data glove, an inexperienced worker is guided through a demonstrative assembly process where each assembly step is documented by the system. Finally, a user test is performed to evaluate the prototype.

Rapid development in processing power, graphic cards and mobile computers open up a wide domain for Mixed Reality applications. Thereby the Mixed Reality continuum covers the complete spectrum from Virtual Reality using immersive projection technology to Augmented Reality using mobile systems like Smartphones and UMPCs. At the Fraunhofer IGD the Mixed Reality Framework instantreality (www.instantreality. org) has been developed as a single and consistent interface for AR/VR developers. This framework provides a comprehensive set of features to support classic Virtual Reality (VR) as well as mobile Augmented Reality (AR). The goal is to provide a very simple application interface which includes the latest research results in the fields of high-realistic rendering, 3D user interaction and total-immersive display technology. The system design is based on various industry standards to facilitate application development and deployment.

Digital twins are increasingly utilized across all lifecycle stages of physical entities. Augmented reality (AR) offers real-time immersion into three-dimensional (3D) data, which provides an immersive experience with dynamic, high-quality, and multi-dimensional digital twins. A robust and customizable data platform is essential to create scalable 3D digital twins; Universal Scene Description (USD) provides these necessary qualities. Given the potential for integrating immersive AR and 3D digital twins, we developed a software application to bridge the gap between multi-modal AR immersion and USD-based digital twins. Our application provides real-time, multi-user AR immersion into USD-based digital twins, making it suitable for time-critical tasks and workflows. AR digital twin software is currently being tested and evaluated in an application we are developing to train astronauts. Our work demonstrates the feasibility of integrating immersive AR with dynamic 3D digital twins. AR-enabled digital twins have the potential to be adopted in various real-time, time-critical, multi-user, and multi-modal workflows.

Biopharma Is Going Digital … Bit by Bit

Research on how to take advantage of Augmented Reality and Virtual Reality applications and technologies in the domain of manufacturing has brought forward a great number of concepts, prototypes, and working systems. Although comprehensive surveys have taken account of the state of the art, the design space of industrial augmented and virtual reality keeps diversifying. We propose a visual approach towards assessing this space and present an interactive, community-driven tool which supports interested researchers and practitioners in gaining an overview of the aforementioned design space. Using such a framework we collected and classified relevant publications in terms of application areas and technology platforms. This tool shall facilitate initial research activities as well as the identification of research opportunities. Thus, we lay the groundwork, forthcoming workshops and discussions shall address the refinement.

Noise monitoring is important in the context of manufacturing because it can help maintain a safe and healthy workspace for employees. Current approaches for noise monitoring in manufacturing are based on acoustic sensors, whose measured sound pressure levels (SPL) are shown as bar/curve charts and acoustic heat maps. In such a way, the noise emission and propagation process is not fully addressed. This paper proposes a digital twin (DT) for noise monitoring in manufacturing using augmented reality (AR) and the phonon tracing method (PTM). In the proposed PTM/AR-based DT, the noise is represented by 3D particles (called phonons) emitting and traversing in a spatial domain. Using a mobile AR device (HoloLens 2), users are able to visualize and interact with the noise emitted by machine tools. To validate the feasibility of the proposed PTM/AR-based DT, two use cases are carried out. The first use case is an offline test, where the noise data from a machine tool are first acquired and used for the implementation of PTM/AR-based DT with different parameter sets. The result of the first use case is the understanding between the AR performance of HoloLens 2 (frame rate) and the setting of the initial number of phonons and sampling frequency. The second use case is an online test to demonstrate the in-situ noise monitoring capability of the proposed PTM/AR-based DT. The result shows that our PTM/AR-based DT is a powerful tool for visualizing and assessing the real-time noise in manufacturing systems.