Lightweight hand and arm tracking for mobile augmented reality

1 IntroductionNowadays, the institutes already use in the process of the advantages of mobile technology, such as portability (the students have easy access at any time to information), the use of digital assets (audio and video files) and huge amount of textual information. This study proposes a new system approach for improving the process by Mobile Augmented Reality. The features of this technology stimulate the environment and consider that the student's motivation is the most important factor in learning.The existing literature [1] highlights two different approaches for augmented reality (AR). Firstly, the authors consider that AR is a system which has the following characteristics: Combines real and virtual; Interactive in real time; Registered in 3-D. Secondly [1] considers Augmented Reality part of Virtual Continuum. This context is composed of two elements: Real Environment - which contains all representations from real life and Virtual Environment - which combines different elements from virtual life. ARs are considered closer to reality than virtual environment because this type of systems are supposed to extend the real objects with virtual elements.According to [2] Augmented Reality is a technology that extends the real world elements with 2D or 3D computer generated objects and lets the users interact with them. Also, the authors present three types of devices which can be used to view the result of AR technology: a Head Mounted Display (HMD), a mobile device - with at least one back camera and a desktop or a laptop computer.Similarly, [2] considers mobile technology as an alternative teaching resource because the users have easy access in real time to information. Also, mobile devices in the process two main problems of traditional are solved: the location and time. Likewise, [1] considers that mobile is a type of eLearning without restrictions of time and space and this technology improves the quality of classical environments, but does not replace them.Other researches, such as [3] and [4] highlight the benefits of a Mobile Augmented Reality system based on tracking objects. In [3] is proposed a mobile augmented reality system based on visual recognition. According to the authors the visual recognition consists into tracking objects identified in real world with the camera of the mobile device and augmenting the scene with virtual objects created by MAR technology. Similarly, [4] presents a Mobile Augmented Reality system bases on 3D registration technology. According to the authors the 3D registration represents the tracking of the position and pose of the mobile device in the real time and real life and finally use these information to insert in the real world the virtual scene created by MAR. The 3D registration technology is based on two type of approaches: markers and natural features. Recently, the 3D registration for MAR uses GPS and sensors from mobile devices to provide different information about position and pose of the camera and combine them with computer vision to improve the speed and accuracy of the system.Other studies, such as [5] propose a makerbased Mobile Augmented Reality solution. The application is used by a Vocational Educational and Training (VET) program for supporting the maintenance process of a car. The authors define VET as education programs that are designed for learners to acquire the knowledge, skills and competencies specific to a particular occupation, trade or class of occupations or trades(p. 49-50). Moreover, in [5] is presented an AR application used by VET students for basic mathematics skills, which is based on videogames.More studies, such as [6] present a Mobile Augmented Reality solution for French language which is based on gamification didactic method. The author, Bernadette Perry, describes gamification as using gamebased mechanics, aesthetics and game thinking to engage people, motivate action, and promote learning (p. …

The boom in the capabilities and features of mobile devices, like smartphones, tablets, and wearables, combined with the ubiquitous and affordable Internet access and the advances in the areas of cooperative networking, computer vision, and mobile cloud computing transformed mobile augmented reality (MAR) from science fiction to a reality. Although mobile devices are more constrained computationalwise from traditional computers, they have a multitude of sensors that can be used to the development of more sophisticated MAR applications and can be assisted from remote servers for the execution of their intensive parts. In this paper, after introducing the reader to the basics of MAR, we present a categorization of the application fields together with some representative examples. Next, we introduce the reader to the user interface and experience in MAR applications and continue with the core system components of the MAR systems. After that, we discuss advances in tracking and registration, since their functionality is crucial to any MAR application and the network connectivity of the devices that run MAR applications together with its importance to the performance of the application. We continue with the importance of data management in MAR systems and the systems performance and sustainability, and before we conclude this survey, we present existing challenging problems.

-In the recent period, many real-world applications and institutions generates a huge amount of data which is unstructured i.e., in the form of images containing data, receipts, invoices, forms, statements, contracts etc. This rich and detailed information presented in the text is of great significance in computer vision-based applications (driverless cars, assisting blind and visually impaired people, detecting labels and packages, automatic number plate recognition etc.). Recently, there has been a hike in the efforts, This project presents AR Weather, a simulation application, which can simulate three types of precipitation: rain, snow, and hail. We examined a range of weather phenomenon and how they may be simulated in a mobile augmented reality system. AR Weather was developed and deployed on the Tinmith wearable computer system to enable autonomous and free movement for the user. The user can move freely inside the simulated weather without limitation. The result of this work, the AR Weather application, has been evaluated with a user study to determine the user’s acceptance and draw conclusions to the applicability of augmented reality simulated weather Keywords- Mobile augmented reality, environmental sensor networks, mobile visualization

With the combination of deep learning and mobile edge computing, the accuracy of the computer vision task in mobile augmented reality (AR) applications can be significantly improved along with the enhancement on the end-to-end latency and energy efficiency. However, no architecture-based delay model for convolutional neural networks (CNNs) has been proposed in edge computing. In this paper, we first develop a new delay model to characterize the relation between the processing delay and the input image size of general CNN models. Then, we formulate a non-convex optimization problem to maximize the learning accuracy under the communication and computation resource constraints. By problem transformation, the optimal resource allocation policy is derived in closed-form and low-complexity search algorithm is also developed. Finally, test results validate the applicability of the delay model and demonstrate the learning accuracy improvement of the proposed algorithm.

A 3-level architecting approach to adaptive resource management in mobile augmented reality systems (MARS) is elaborated, which is based on comprehensive data structuring and analyzing of their specific hard- and software features. At the conceptual modeling level an ontology of adaptive MARS resources is constructed, and at the logical modeling level a generic algorithmic model is proposed, which can be instantiated in the collection of specific methods and metrics. As a physical model the reference software architecture for adaptive resource management in MARS is designed, and this approach is implemented partly as a software prototype. It is tested successfully to solve the task of adaptive image resolution on mobile device, according to changes of computational load that finally enables better video stream quality in MARS.

Augmented reality simulations aims to provide realistic blending between real world and virtual objects. One of the important factors for realistic augmented reality is correct illumination simulation. Mobile augmented reality systems is one of the best options for introducing augmented reality to the mass market due to its low production cost and ubiquitousness. In mobile augmented reality systems, the ability to correctly simulate in realtime the illumination direction that matches the illumination direction of the real world is limited. Developing a mobile augmented reality systems with the ability to estimate illumination direction presents a challenge due to low computation power and dynamically changing environment. In this paper, we described a new method that we have developed for realtime illumination direction estimation for mobile augmented reality systems, using analysis of shadow produced by a reference object that doubles as a 3D augmented reality marker. The implementation of the method could estimate the direction of a single strong light source in a controlled environment with a very good degree of accuracy, with angular error averaging lower than 0.038 radians. The current implementation achieved 2.1 FPS performance in a low-end Android mobile device, produced proper estimation within 15 seconds using a uniform surface, and demonstrated scalability potential.

The advancement in deep learning and edge computing has enabled intelligent mobile augmented reality (MAR) on resource limited mobile devices. However, today very few deep learning based MAR applications are applied in mobile devices because they are significantly energy-guzzling. In this paper, we design a user preference based energy-aware edge-based MAR system that enables MAR clients to dynamically change their configuration parameters, such as CPU frequency and computation model size, based on their user preferences, camera sampling rates, and available radio resources at the edge server. Our proposed dynamic MAR configuration adaptations can minimize the per frame energy consumption of multiple MAR clients without degrading their preferred MAR performance metrics, such as service latency and detection accuracy. To thoroughly analyze the interactions among MAR configuration parameters, user preferences, camera sampling rate, and per frame energy consumption, we propose, to the best of our knowledge, the first comprehensive analytical energy model for MAR clients. Based on the proposed analytical model, we develop a LEAF optimization algorithm to guide the MAR configuration adaptation and server radio resource allocation. Extensive evaluations are conducted to validate the performance of the proposed analytical model and LEAF algorithm.

With advances in tracking and increased computing power, mobile AR systems are popular in our daily life. Researchers in mobile AR technology have emphasized the technical challenges involved in the limitations imposed from mobility. They did not consider context-aware service with user-related information annotation, even if in ubiquitous computing environment, various contexts of both a user and an environment can be utilized easily as well as effectively. Moreover, it is difficult to have access to pervasive but invisible computing resources. At the same time, the more smart appliances become evolved with more features, the harder their user interfaces tend to become to use. Thus, in this paper, we propose Context-aware Mobile Augmented Reality (CaMAR) system. It lets users interact with their smart objects through personalized control interfaces on their mobile AR devices. Also, it supports enabling contents to be not only personalized but also shared selectively and interactively among user communities.

This paper presents MARVEL, a mobile augmented reality (MAR) system which provides a notation display service with imperceptible latency (<100 ms) and low energy consumption on regular mobile devices. In contrast to conventional MAR systems, which recognize objects using image-based computations performed in the cloud, MARVEL mainly utilizes a mobile device's local inertial sensors for recognizing and tracking multiple objects, while computing local optical flow and offloading images only when necessary. We propose a system architecture which uses local inertial tracking, local optical flow, and visual tracking in the cloud synergistically. On top of that, we investigate how to minimize the overhead for image computation and offloading. We have implemented and deployed a holistic prototype system in a commercial building and evaluate MARVEL's performance. The efficient use of a mobile device's capabilities lowers latency and energy consumption without sacrificing accuracy.

The purpose of this study was to investigate how a mobile collaborative augmented reality (AR) system affects learnersi¦ knowledge construction behaviors and learning performances. In this study, 20 undergraduate students were recruited and divided into dyads to discuss given questions with the assistance of mobile collaborative AR system named AR Physics. The participantsi¦ knowledge regarding elastic collision was evaluated through a pretest and a posttest that occurred right after the treatment. Learnersi¦ knowledge construction behaviors were qualitatively identified according to a selected coding scheme and then were analyzed by adopting sequential analysis. The results indicated that learners significantly gained their knowledge on the topic of elastic collision after completing the given task by manipulating AR Physics system. Furthermore, sequential patterns of learnersi¦ behaviors during knowledge construction activities were identified.

This article presents a mobile Augmented Reality system, called DiedricAR, aimed at the learning of Descriptive Geometry. Thanks to its ability to recreate virtual models in real space, Augmented Reality is a technology suitable for making Descriptive Geometry comprehension and interpretation easier. The DiedricAR application allows students to learn in autonomously way by using their own mobile devices (smartphones and tablets), that work as Augmented Reality displays over training material (DiedricAR exercise workbook) specially designed for the new learning model defined by the European Higher Education System. Compared to some of the existing Augmented Reality systems used to learn Descriptive Geometry, DiedricAR offers the advantage of being specifically developed for mobile devices giving the students the possibility of using ubiquitous learning to its ultimate extent by interacting with the didactical content (i.e. showing the desired intermediate step when solving dihedral exercises). The presentation of DiedricAR is completed by exploring some key items such as the potential benefits for students' spatial ability, the relationship between application design and user experience, and software performance on several mobile devices.

This paper presents two visual registration solutions for a mobile augmented reality system. The first one is a marker based solution whereas the second one is a hybrid approach. The hybrid method combines a coded marker technique for the initialization in the first frame, and a markerless registration in the next frames thanks to a 3-D model based tracking method. Because this mobile augmented reality system is designed for use in the industrial context of maintenance assistance for instance- robustness, accuracy, real-time and user comfort are the main concerns. For the different stages of the proposed solutions, various algorithms were evaluated to determine which one offers the best robustness and efficiency.KeywordsAugmented RealityReprojection ErrorMobile Augmented RealityCollaborative Augmented RealityInter Frame MotionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This paper presents a new context-aware mobile augmented reality system that provides rapid and robust on-site access to construction project information such as drawings, specifications, schedules, and budgets. The mobile augmented reality system does not need any RF-based location tracking (e.g., GPS or WLAN) or optical fiducial markers for tracking a user’s position. Rather, the user’s location and orientation are automatically and purely derived by comparing photographs from the user’s phone to a 3D point cloud model created from a set of site photographs. After generating a 3D point cloud model of construction site, field personnel can use mobile devices to take pictures of building elements and be presented on-site with a detailed list of project information related to the visible construction elements in an augmented reality format. The experimental results show that (1) the underlying 3D reconstruction module of the system generates more complete 3D point cloud models, and faster than other state-of-the-art Structure-from-Motion(SfM) algorithms; (2) the localization method is an order of magnitude more accurate than the state-of-the-art solutions, and can provide acceptable tolerances of most on-site engineering applications. Using an actual construction case study, the perceived benefits and limitations of the proposed method for on-site context-aware applications are discussed in detail.

The study of materials science concepts, such as crystallographic structures, poses well‐known pedagogical challenges due to the spatial visualization demands imposed by traditional two‐dimensional representations. To address these challenges, this paper presents CeldApp , a low‐cost mobile system that integrates interactive visualization and augmented reality to support the learning of core materials science topics. The system provides a set of modular 3D learning environments covering crystallization processes, crystal structures, and phase diagram analysis, enabling students to actively explore abstract concepts through direct manipulation and immediate visual feedback. The educational impact of the proposed system was investigated through a controlled pilot study employing a pretest–posttest design with parallel groups in an undergraduate materials science course. Learning gain, task completion time, and user experience were considered as evaluation measures. While between‐group differences in learning gain did not reach conventional levels of statistical significance, the experimental group exhibited a consistent positive trend with a moderate effect size. Notably, participants using the system demonstrated a statistically significant reduction in task completion time, suggesting improved efficiency in knowledge retrieval and problem‐solving. Qualitative feedback further indicated high perceived usability and strong educational value of the augmented reality components. Taken together, these findings suggest that accessible, low‐cost immersive visualization tools hold promise for mitigating spatial visualization challenges in materials science education and motivate further large‐scale investigation.

This paper proposes a mobile augmented reality system that can model 3D virtual objects and author augmented reality contents on site. The differences of the proposed system from the existing ones are an interaction approaches used to generate and manipulate primitives and additional features such as a shadow and a multi-freezing mode to create realistic augmented reality contents efficiently.