SADNet: Generating Immersive Virtual Reality Avatars by Real-time Monocular Pose Estimation

Recent advances in extended reality (XR) technology have opened the possibility of significantly improving telemedicine systems. This is primarily achieved by transferring 3D information about patient state, which is utilized to create more immersive experiences on VR/AR headsets. In this paper, we propose an XR-based telemedicine collaboration system in which the patient is represented as a 3D avatar in an XR space shared by local and remote clinicians. The proposed system consists of an AR client application running on Microsoft HoloLens 2 used by a local clinician, a VR client application running on the HTC vive Pro used by a remote clinician, and a backend part running on the server. The patient is captured by a camera on the AR side, and the 3D body pose estimation is performed on frames from this camera stream to form a 3D patient avatar. Additionally, the AR and VR sides can interact with the patient avatar via virtual hands, and annotations can be performed on a 3D model. The main contribution of our work is the use of 3D body pose estimation for the creation of a 3D patient avatar. In this way, 3D body reconstruction using depth cameras is avoided, which reduces system complexity and hardware and network resources. Another contribution is the novel architecture of the proposed system, where audio and video streaming are realized using WebRTC protocol. The performance evaluation showed that the proposed system ensures high frame rates for both AR and VR client applications, while the processing latency remains at an acceptable level.

This study introduces an innovative framework for neurological rehabilitation by integrating brain-computer interfaces (BCI) and virtual reality (VR) technologies with the customization of three-dimensional (3D) avatars. Traditional approaches to rehabilitation often fail to fully engage patients, primarily due to their inability to provide a deeply immersive and interactive experience. This research endeavors to fill this gap by utilizing motor imagery (MI) techniques, where participants visualize physical movements without actual execution. This method capitalizes on the brain's neural mechanisms, activating areas involved in movement execution when imagining movements, thereby facilitating the recovery process. The integration of VR's immersive capabilities with the precision of electroencephalography (EEG) to capture and interpret brain activity associated with imagined movements forms the core of this system. Digital Twins in the form of personalized 3D avatars are employed to significantly enhance the sense of immersion within the virtual environment. This heightened sense of embodiment is crucial for effective rehabilitation, aiming to bolster the connection between the patient and their virtual counterpart. By doing so, the system not only aims to improve motor imagery performance but also seeks to provide a more engaging and efficacious rehabilitation experience. Through the real-time application of BCI, the system allows for the direct translation of imagined movements into virtual actions performed by the 3D avatar, offering immediate feedback to the user. This feedback loop is essential for reinforcing the neural pathways involved in motor control and recovery. The ultimate goal of the developed system is to significantly enhance the effectiveness of motor imagery exercises by making them more interactive and responsive to the user's cognitive processes, thereby paving a new path in the field of neurological rehabilitation.

Despite the progress of 3D Gaussian Splatting (3DGS), reconstructing one-shot animatable 3D human avatars from a single image remains a challenging task. Existing 3DGS-based methods primarily rely on appearance observation and motion cues from monocular videos to reconstruct animatable 3D avatars. However, when applied to single-image setups, these methods struggle to extract accurate 3D features from a 2D image, limiting their ability to capture fine-grained appearance details and dynamic deformation, especially from challenging perspectives. In this work, we propose SIAvatar, a novel single-image human reconstruction method that integrates diffusion-based appearance prior and parametric human model geometry prior within a 3DGS framework. Secondly, a vertex-based adaptive Gaussian densification scheme is introduced to effectively represent human geometry while mitigating artifacts. Extensive experiments demonstrate that SIAvatar generates realistic 3D avatars with plausible appearance details and novel pose animation from a single input image. Project page: https://siavatar.github.io/.

Shared control of avatars in virtual reality is emerging as a promising method to enhance collaborative interactions. While the literature investigates shared control as an input modality to understand the design of functional experiences, the user experience associated with shared control remains underexplored. In this paper, we investigate the user experience of shared control using a networked VR environment, "CoplayingVR". Through a user study comprised of three gaming tasks in VR with 48 participants, we collect both quantitative and qualitative data on the user experience and performance of shared control. Our findings reveal that the shared control mode significantly improved user experience factors and brought novelty to the play. Furthermore, shared control significantly enhanced the task efficiency for novice users. Finally, we discuss how our findings inform the implications for research and practice of shared control for the growing body of research in VR.

This paper presents a different, until now, perspective of aesthetic experience during the process of designing 3D avatars, formulating and visualizing the combination of user-experience and psychology. The present research aims to define 3D aesthetic experience and the relation of HCI Human-Computer Interaction through a theoretical model delivering new insights on the process of 3D avatars' design. There is limited research about the procedure and the influences emotions, mood and external factors during the process of users designing their three-dimensional 3D depictions otherwise known as avatars. The provided theoretical model is a combination of the three subjective factors aesthetics, psychology and user experience which are interrelated, and are present during such design procedures. The common element that connects all of these areas is Human Computer Interaction HCI. In each area of interest, on its own, there is relevant and sufficient research. But, to a large extent their common relation to 3D environments has not been explored yet. It is the objective of this paper to explore the aesthetic experience from the designer view, in other words, is defined the 3D aesthetic experience of the user but from the side of the designer. An additional reason of the formulation and the focus on 3D visualization is to simplify the procedure of 3D avatar design while simultaneously embracing the influence of aesthetics, user experience and psychology; which are provided via an overview of existing research, concentrating on the procedure for 3D avatar design.

The embodiment of avatars in virtual reality (VR) is a promising tool for enhancing the user’s mental health. A great example is the treatment of body image disturbances, where eliciting a full-body illusion can help identify, visualize, and modulate persisting misperceptions. Augmented reality (AR) could complement recent advances in the field by incorporating real elements, such as the therapist or the user’s real body, into therapeutic scenarios. However, research on the use of AR in this context is very sparse. Therefore, we present a holographic AR mirror system based on an optical see-through (OST) device and markerless body tracking, collect valuable qualitative feedback regarding its user experience, and compare quantitative results regarding presence, embodiment, and body weight perception to similar systems using video see-through (VST) AR and VR. For our OST AR system, a total of 27 normal-weight female participants provided predominantly positive feedback on display properties (field of view, luminosity, and transparency of virtual objects), body tracking, and the perception of the avatar’s appearance and movements. In the quantitative comparison to the VST AR and VR systems, participants reported significantly lower feelings of presence, while they estimated the body weight of the generic avatar significantly higher when using our OST AR system. For virtual body ownership and agency, we found only partially significant differences. In summary, our study shows the general applicability of OST AR in the given context offering huge potential in future therapeutic scenarios. However, the comparative evaluation between OST AR, VST AR, and VR also revealed significant differences in relevant measures. Future work is mandatory to corroborate our findings and to classify the significance in a therapeutic context.

There has been a continued trend towards minimizing instrumentation for full-body motion capture, going from specialized rooms and equipment, to arrays of worn sensors and recently sparse inertial pose capture methods. However, as these techniques migrate towards lower-fidelity IMUs on ubiquitous commodity devices, like phones, watches, and earbuds, challenges arise including compromised online performance, temporal consistency, and loss of global translation due to sensor noise and drift. Addressing these challenges, we introduce MobilePoser, a real-time system for full-body pose and global translation estimation using any available subset of IMUs already present in these consumer devices. MobilePoser employs a multi-stage deep neural network for kinematic pose estimation followed by a physics-based motion optimizer, achieving state-of-the-art accuracy while remaining lightweight. We conclude with a series of demonstrative applications to illustrate the unique potential of MobilePoser across a variety of fields, such as health and wellness, gaming, and indoor navigation to name a few.

The imagination is a powerful social and revolutionary force. In the following discussion, we consider the health benefits of imagination and the role of immersive virtual experiences, such as virtual reality (VR), in facilitating imagination, particularly for Black youth. The paramount focus on understanding and attenuating threats that digital media pose to health should be complemented by research examining ways digital media may enrich the lives of young people. Immersive technologies have the potential to fundamentally change the ways we observe, learn, think, engage, and imagine. We provide an overview and critique of racial embodiment in VR and the sparse empirical literature linking health, medicine, and imagination. The concepts of radical and untethered imagination are also introduced as distinct imaginary processes that may promote health and be facilitated through the use of VR. Recommendations for future research linking imagination and health are also provided.

To make 3D human avatars widely available, we must be able to generate a variety of 3D virtual humans with varied identities and shapes in arbitrary poses. This task is chal-lenging due to the diversity of clothed body shapes, their complex articulations, and the resulting rich, yet stochas-tic geometric detail in clothing. Hence, current methods that represent 3D people do not provide a full generative model of people in clothing. In this paper, we propose a novel method that learns to generate detailed 3D shapes of people in a variety of garments with corresponding skin-ning weights. Specifically, we devise a multi-subject forward skinning module that is learned from only a few posed, unrigged scans per subject. To capture the stochastic nature of high-frequency details in garments, we leverage an adversarial loss formulation that encourages the model to capture the underlying statistics. We provide empirical evi-dence that this leads to realistic generation of local details such as wrinkles. We show that our model is able to gen-erate natural human avatars wearing diverse and detailed clothing. Furthermore, we show that our method can be used on the task of fitting human models to raw scans, out-performing the previous state-of-the-art.

Animated 3D human avatars from a single image with GAN-based texture inference

Human detection and pose estimation are essential for understanding human activities in images and videos. Mainstream multi-human pose estimation methods take a top-down approach, where human detection is first performed, then each detected person bounding box is fed into a pose estimation network. This top-down approach suffers from the early commitment of initial detections in crowded scenes and other cases with ambiguities or occlusions, leading to pose estimation failures. We propose the DetPoseNet, an end-to-end multi-human detection and pose estimation framework in a unified three-stage network. Our method consists of a coarse-pose proposal extraction sub-net, a coarse-pose based proposal filtering module, and a multi-scale pose refinement sub-net. The coarse-pose proposal sub-net extracts whole-body bounding boxes and body keypoint proposals in a single shot. The coarse-pose filtering step based on the person and keypoint proposals can effectively rule out unlikely detections, thus improving subsequent processing. The pose refinement sub-net performs cascaded pose estimation on each refined proposal region. Multi-scale supervision and multi-scale regression are used in the pose refinement sub-net to simultaneously strengthen context feature learning. Structure-aware loss and keypoint masking are applied to further improve the pose refinement robustness. Our framework is flexible to accept most existing top-down pose estimators as the role of the pose refinement sub-net in our approach. Experiments on COCO and OCHuman datasets demonstrate the effectiveness of the proposed framework. The proposed method is computationally efficient (5-6x speedup) in estimating multi-person poses with refined bounding boxes in sub-seconds.

6D pose estimation is an important branch in the field of vision measurement, and is widely used in the fields of robotics, autonomous driving and reality augmentation. The latest research trend in 6D pose estimation is to train a deep neural network to directly predict the 2D projection position of the 3D keypoint from the image, establish the corresponding relationship, and finally use the perspective-n-point (PnP) algorithm to perform pose estimation. The current challenge of pose estimation is that when objects are textureless, occluded or scene-clutterd, the detection accuracy is reduced, and most of the existing algorithm models are large and cannot accommodate real-time requirements. In this paper, we introduce a densely connected feature pyramid network (DFPN) that can efficiently integrate and utilize features. We combine the cross-stage partial network (CSPNet) with DFPN to design a new network for 6D pose estimation, DFPN-6D, a new approach for 6D object pose estimation. DFPN-6D can efficiently and accurately handle objects with textureless, occluded and scene clutter and estimate their full 6D poses in a single shot. Furthermore, we propose a new confidence calculation method and loss function for object pose estimation, which can fully consider spatial information. Finally, we propose a novel augmentation method for direct 6D pose estimation approaches to improve performance and generalization ability in the case of occlusion, which is called 6D augmentation. Our approach achieves a new state-of-the-art accuracy of 98.06 and 87.09 in terms of the ADD(-S) metric on the Linemod dataset and Occluded-Linemod dataset, and our method also achieves the best result in terms of the different metric on the MULT-I dataset, BIN-P dataset and T-LESS dataset, respectively, while still running end-to-end at over 65 FPS. The experimental results demonstrate that our algorithm is robust to textureless materials and occlusion while running more efficiently than other methods. We also deploy our proposed method to a real robot to grasp and manipulate objects based on the estimated pose.

Backgrounds. Virtual reality (VR) simulates real-life events and scenarios and is widely utilized in education, entertainment, and medicine. VR can be presented in two dimensions (2D) or three dimensions (3D), with 3D VR offering a more realistic and immersive experience. Previous research has shown that electroencephalogram (EEG) profiles induced by 3D VR differ from those of 2D VR in various aspects, including brain rhythm power, activation, and functional connectivity. However, studies focused on classifying EEG in 2D and 3D VR contexts remain limited.Methods. A 56-channel EEG was recorded while visual stimuli were presented in 2D and 3D VR. The recorded EEG signals were classified using two machine learning approaches: traditional machine learning and deep learning. In the traditional approach, features such as power spectral density (PSD) and common spatial patterns (CSP) were extracted, and three classifiers-support vector machines (SVM), K-nearest neighbors (KNN), and random forests (RF)-were used. For the deep learning approach, a specialized convolutional neural network, EEGNet, was employed. The classification performance of these methods was then compared.Results. In terms of accuracy, precision, recall, and F1-score, the deep learning method outperformed traditional machine learning approaches. Specifically, the classification accuracy using the EEGNet deep learning model reached up to 97.86%.Conclusions. EEGNet-based deep learning significantly outperforms conventional machine learning methods in classifying EEG signals induced by 2D and 3D VR. Given EEGNet's design for EEG-based brain-computer interfaces (BCI), this superior classification performance suggests that it can enhance the application of 3D VR in BCI systems.

3D keypoint is widely used for object recognition and pose estimation with a 3D point cloud since it has robustness against measurement noise and occlusion. Many types of 3D keypoints and the corresponding detection methods of 3D keypoints have been proposed. It is essentially important to select suitable 3D keypoint depending on a target object, scene situation, and 3D measurement system. While there have been a lot of useful methods for detecting 3D keypoints, detecting time is an issue, especially for real-time applications, such as robot vision. The detecting of 3D keypoints tends to entail a trade-off between computation time and the robustness of the 3D keypoint. To solve these problems, we propose a FAst Detection Algorithm for 3D keypoints named FADA-3K. A user can select one of the suitable 3D keypoints and the corresponding detection method which has been already proposed since FADA-3K can be implemented as an add-on of the existing detection methods. Numerical experiments show that FADA-3K can achieve about nine times faster detection than conventional approaches to detecting 3D keypoints.

Providing Virtual Reality(VR) users with a 3D representation of their body complements the experience of immersion and presence in the virtual world with the experience of being physically located and more personally involved. A full-body avatar representation is known to induce a Sense of Embodiment (SoE) for this virtual body, which is associated with improvements in task performance, motivation and motor learning. Recent experimental research on embodiment provides useful guidelines, indicating the extent of discrepancy tolerated by users and, conversely, the limits and disruptive events that lead to a break in embodiment (BiE). Based on previous works on the limit of agency under movement distortion, this paper describes, studies and analyses the impact of a very common yet overlooked embodiment limitation linked to articular limits when performing a reaching movement. We demonstrate that perceiving the articular limit when fully extending the arm provides users with an additional internal proprioceptive feedback which, if not matched in the avatar’s movement, leads to the disruptive realization of an incorrect posture mapping. This study complements previous works on self-contact and visuo-haptic conflicts and emphasizes the risk of disrupting the SoE when distorting users’ movements or using a poorly-calibrated avatar.