Abstract Skeleton‐based action recognition has been continuously and intensively studied. However, dynamic 3D skeleton data are difficult to be popularized in practical applications due to the restricted data acquisition conditions. Although the action recognition method based on 2D pose information extracted from RGB video can effectively avoid the influence of complex background, it is susceptible to factors such as video jitter and joint overlap. To reduce the interference of the aforementioned factors, we use two‐dimensional skeletal joint coordinate modal information to represent the changes in human body posture. First, we use a target detector and pose estimation algorithm to obtain the joint coordinates of each frame sample from RGB video. Then the feature extraction network is combined to perform multi‐level feature learning to establish correspondence between actions and corresponding multi‐level features. Finally, the hierarchical attention mechanism is introduced to design the model named CHAN. By calculating the association between elements, the weight of the action classification is redistributed. Extensive experiments on three datasets demonstrate the effectiveness of our proposed method.

Abstract This article presents a reinforcement learning‐based approach to generate animation in which two agents use a rope to collaboratively transport a block. The challenge is that the agents need to master several skills, including approaching the block, using the rope to wrap around it, and then moving the block to a predefined goal position. We propose several reward terms to learn the transportation policy and the adjustment policy that govern the skills of the agents. Experiment results showed that the proposed approach was able to generate various animations in different settings, including rope lengths, block sizes, and block shapes. An ablation test revealed the effects of the reward terms. We also investigated factors that affected the performance of the two policies.

SummaryThis paper focuses on understanding how study participants interact and perceive a virtual crowd in an immersive virtual environment. Specifically, our within‐group exploratory study investigated how avoidance proximity variations (i.e., low, medium, and high avoidance proximity [defined as avoidance radius]) assigned to crowd agents impacted participants' interaction with the virtual crowd. During the study, we instructed our participants to walk in a virtual environment. At the same time, we had a virtual crowd scripted to walk toward the start position of the participant following a straight path. During the participants' walking task, we collected movement data (i.e., trajectory length and completion time) and immediately after each experimental condition, we asked participants to self‐report their experience (i.e., co‐presence, behavioral independence, crowd realism, crowd interaction realism, perceived politeness, and emotional reactivity). Based on the collected data, we found that when we exposed our participants to the high avoidance proximity condition, they: (1) followed longer paths, (2) spent more time reaching the target goal, (3) rated the virtual crowd less polite, (4) rated the virtual crowd and their interaction with the virtual crowd less realistic, (5) rated the behavior independence of the virtual crowd lower, and (6) self‐reported higher emotional reactivity. We discuss our findings and suggestions for further research on human‐virtual crowd interaction.

Abstract Industrial sectors are reinventing in automation, stability, and robustness due to the rapid development of artificial intelligence technologies, resulting in significant increases in quality and production. Visual‐based sensor networks capture various views of the surrounding environment and are used to monitor industrial and transportation sectors. However, due to unclean suspended air particles that damage the whole monitoring and transportation systems, the visual quality of the images is degraded under adverse weather conditions. This research proposed a convolutional neural network‐based image dehazing and detection approach, called end to end dehaze and detection network (EDD‐N), for proper image visualization and detection. This network is trained on real‐time hazy images that are directly used to recover dehaze images without a transmission map. EDD‐N is robust, and accuracy is higher than any other proposed model. Finally, we conducted extensive experiments using real‐time foggy images. The quantitative and qualitative evaluations of the hazy dataset verify the proposed method's superiority over other dehazing methods. Moreover, the proposed method validated real‐time object detection tasks in adverse weather conditions and improved the intelligent transportation system.

Abstract We propose a novel method to avoid stereo window violations at screen borders. These occur for objects in front of the zero parallax plane, which appear in front of the (physical) screen, and that are clipped for one eye while still being visible for the other eye. This contradicts other stereo cues, particularly disparity, potentially resulting in eye strain and simulator sickness. In interactive and dynamic virtual environments, where the user controls the camera, for example, via head tracking, it is impossible to avoid stereo window violations completely. We propose Dynparity, a novel rendering method to eliminate the conflict between clipping and negative disparity, by introducing a nonuniform stereoscopic projection. For each vertex in front of the zero parallax plane, we compute the stereoscopic projection such that the parallax approaches zero toward the edge of the screen. Our approach works entirely on the GPU in real‐time and can be easily included in modern game engines. We conducted a user study comparing our method to the standard stereo projection on a large‐screen stereo wall with head tracking. Our results show significantly reduced simulator sickness when using Dynparity compared to the standard stereo rendering.

Abstract While visual saliency has been used for various purposes in virtual reality (VR), the efforts to properly understand the saliency mechanism in VR remain insufficient. In this paper, we present an extensive comparative analysis of learning‐based and heuristic‐based approaches to visual saliency prediction in immersive VR experienced using head‐mounted‐displays with a particular focus on the contribution of the depth cue. To this end, we use three learning‐based RGB‐D image saliency detection methods and two heuristic‐based RGB‐D image saliency detection methods on a VR dataset curated from three distinct virtual environments under two‐dimensional and three‐dimensional viewing conditions. Additionally, we extend the analysis by including a heuristic‐based RGB video saliency detection method and its depth‐infused version. The results acquired using these seven methods reveal the superiority of the learning‐based RGB‐D image saliency prediction methods in VR and validate the importance of the depth cue in the saliency prediction of virtual environments.

Abstract Locomotion techniques are an important part of the virtual reality (VR) experience, as they enable a means of movement within simulated computer graphics. Studies have shown that locomotion techniques cause unpleasant effects related to cybersickness. In this article, we report on the results of a user study (N = 29) aimed to investigate and compare three types of locomotion techniques in VR in terms of their impact on cybersickness. The different locomotion techniques are tested in two different contrast scene settings, daytime and nighttime, to further explore the potential impact of scene contrast adjustments. For the evaluation of cybersickness, we used a questionnaire to obtain subjective ratings, and heart rate monitoring as an objective metric. Results show that a linear movement locomotion technique provokes the highest level of cybersickness, and that women have a higher heart rate as compared to men when navigating and interacting in a VR scene. Regarding the influence of scene contrast, results showed that scenes with daylight were better suited to participants in almost all tested scenarios. In addition to reported findings related to locomotion techniques and the impact on cybersickness, we highlight that a key contribution is the utilized test methodology.

Abstract Multi‐agent simulation has attracted much attention in the field of computer animation in the last decades for its ability to model interaction between autonomous micro level entities. It widely uses deep reinforcement learning (DRL), which allows us to model environments and its agents approaching real‐world and human‐level complexity, with applications in robotics and computer animation, among others. However, DRL multi‐agent simulation faces additional challenges: they have to be able to generalize high‐dimensional observations and relate them with a high‐dimensional action space, maximizing long‐term cumulated reward. Due to this, DRL systems with numerous interacting agents seldom consider skeleton‐level action spaces. To this end, we present skeleton‐level control for multi‐agent simulation with DRL. With our method, we are able to procedurally generate real‐time collision‐free simulations directly on individual agents with a high‐dimensional skeleton‐level action space. The state in our DRL system includes the velocity of the agent, its destination, and the status of its joints, as well as visual‐based information about the environment and other agents. Our reward function encourages motion into the target destinations, and enalizes collision. We provide extensive experimentation to show the ability of agents to reach their goal through its skeleton motion while successfully avoiding inter‐collisions.

Abstract Single image 3D model retrieval has attracted a lot of attentions with the convenience of organizing large‐scale unlabeled 3D models. Existing methods transfer the knowledge from well‐annotated 2D images (i.e., source domain) to unlabeled 3D models (i.e., target domain) to improve the discriminability of 3D models and align the feature distributions of 2D images and 3D models. However, during the alignment, the feature learning target of improving the discriminability of 3D models sometimes confuses the boundaries between 2D image categories, where prior methods ignore keeping the discriminability of 2D images. Motivated by this observation, we propose a source‐enhanced prototypical alignment framework to first remain the discriminability of 2D images and then guide the category‐level cross‐domain alignment with better image representations. Specifically, a novel separation and compactness loss is proposed for images to separate the samples from different categories and compact the samples within the same category. Then we perform prototypical alignment to make 2D image features assist in the discriminative feature learning for 3D models. We evaluate the proposed method on the commonly used cross‐domain 3D model retrieval benchmarks, namely MI3DOR and MI3DOR‐2, and the results demonstrate the effectiveness of the proposed method.

Abstract We propose a muscle‐driven motion generation approach to realize virtual human motion with user interaction and higher fidelity, which can address the problem that the joint‐driven fails to reflect the motion process of the human body. First, a simplified virtual human musculoskeletal model is built based on human biomechanics. Then, a hierarchical policy learning framework is constructed including motion tracking layer, SPD controller and muscle control layer. The motion tracking layer is responsible for mimicking reference motion and completing control command, using proximal policy optimization to train the policy; the muscle control layer is aimed to minimize muscle energy consumption and train the policy based on supervised learning; the SPD controller acts as a link between the two layers. At the same time, we integrate the curriculum learning to improve the efficiency and success rate of policy training. Simulation experiments show that the proposed approach can use motion capture data and pose estimation data as reference motions to generate better and more adaptable motions. Furthermore, the virtual human has the ability to respond to the user control command during the motion, and can complete the target task successfully.