Muscle‐driven virtual human motion generation approach based on deep reinforcement learning

Maintainability is one of the main targets of product development. Through simulating the actual maintenance process, maintainability problems can be identified by virtual reality technology. Most virtual reality softwares provide virtual human motion simulation functions, but the virtual human motion control methods of them are defective. In order to enhance virtual human motion control, maintenance task model was established in this paper, virtual human motion model was designed. Then a virtual human motion database based on Jack was developed by Tcl/Tk Python language, and it was applied to a real maintenance process. Furthermore, the maintenance process was simulated in Jack and the ergonomics of it were analyzed. The result showed that the motion database can be effectively applied to virtual maintenance simulation and make the maintenance process-related ergonomics analysis conveniently.

This survey firstly reviews motion capture technology and the current challenge associated with the application in virtual human's control area. Secondly simplifies Jack's joint hierarchical model in order to drive Jack's virtual human to move by motion capture data. Thirdly uses Euler angle rotation equation to achieve pose mapping between BVH and Jack. Finally, results from experimentation show that when the motion capture data contained rich human motion details are retargeted to Jack by the paper's algorithms, the real-time interaction between virtual human and environment seems more real.

In the complex landscape of educational information, the integration of Virtual Reality (VR) technology into secondary school physical education presents a promising avenue to enhance both teaching quality and efficiency. This article delineates the construction of a VR-based smart sports classroom, focusing on the overarching structure of the teaching program and the pivotal teaching processes. It discusses methods for analyzing and editing sports teaching videos and details the integration of VR with sports posture estimation. This includes the development of a virtual human body motion imaging model and corresponding motion-solving algorithms. An empirical study was conducted within a secondary school sports classroom to assess the feasibility and impact of VR technology on physical education. The study involved a comparative analysis of physical quality between an experimental group taught using VR technology and a control group taught using traditional methods. Additionally, the study examined changes in learning interest pre- and post-intervention in both groups. Statistical analysis revealed that the p-values for the three evaluated physical tests—20-meter sprint, shuttle run, and 30-second rope skipping—were all above 0.05, suggesting no significant difference in physical performance due to the VR intervention. However, significant improvements were observed in the dimensions of sports participation (p < 0.05), positive interest (p = 0.013 < 0.05), and focus on sports (p > 0.05), indicating an enhancement in students’ engagement and attitudes toward learning. Conversely, the negative interest dimension also showed significant changes (p < 0.05). These findings underscore the beneficial role of VR technology in enriching the learning experience and potentially improving academic outcomes in college-level physical education.

There have been some biomechanics-based control systems that have achieved better realistic virtual human motion. Yet their abilities to adapt the changing environments are weaker than the traditional control systems with characters driven by proportional derivative actuators directly. In our method, we build a hierarchical neuromuscular virtual human (NMVH) motion control system that consists of a low-level spine reflex layer and a high-level policy control layer. The spine reflex layer uses a feedback net to map sensory information to excitations, which stimulate muscles to generate joint torques. The policy control layer includes a deep neural network, which provides a learned action policy to spine reflex layer for achieving terrain-adaptive motion skills. The particle swarm optimization algorithm is used to optimize the gain factors of the feedback net for finding out a basic policy to make the virtual human walk on the flat terrain autonomously. The proximal policy optimization algorithm is employed to train the deep neural network in policy control layer for learning how to modulate the actions to adapt to the changing terrain. The simulation results in Matlab show that virtual human can walk smoothly and better adapt to the given terrain changes. It demonstrates that our control system improves the terrain-adaptive walking skill of the neuromuscular virtual human.

Aiming at the problems existing in traditional upper limb rehabilitation training, such as low rehabilitation efficiency, single and passive training methods, etc. Deep reinforcement learning can provide a better trajectory planning scheme for the upper limb rehabilitation robot working in multi-dynamic environment. It can make more efficient and active rehabilitation training plan for rehabilitation patients. In the actual rehabilitation training, it is necessary to smooth the motion track without mutation and avoid the singular region. In this paper, DDPG deep learning reinforcement learning algorithm is used to propose an improved reward function to provide global guidance for exploring optimal trajectory planning. At the same time, the nonlinear function of the feedback regulation mechanism is constructed by dividing the working interval property to accelerate the exploration process and avoid the singular region. Experiments show that the convergence of the reward value of the improved model is significantly improved compared with the traditional model. The method proposed in this paper provides a good theoretical basis for the research of active rehabilitation control system of upper limb rehabilitation robot.

<div>When the vehicle system performs trajectory tracking control, it presents relatively complex nonlinear coupling dynamics characteristics. The traditional coordination algorithm relying on a simplified linear model is mostly unable to deal well with the actual nonlinear dynamic behaviors. In contrast, reinforcement learning (RL) method will derive the optimal strategy by means of interaction with the environment. This eliminates the need for accurate vehicle modeling. These methods use all of the nonlinear approximation capabilities of deep neural networks and can effectively reflect the complex relationship between vehicle state and control actions. The framework itself supports multidimensional input processing and continuous operation space optimization because of the development of parallel processing architectures. In order to reduce the motion jitter caused by the direct generation of front and rear wheel angles by the network, this article uses steering angle increments as control commands and effectively reduces motion jitter through differential action formulas. To accelerate policy convergence, behavioral cloning (BC) technology was integrated into the deep reinforcement learning (DRL) training pipeline and initial policy parameterization was performed using expert demonstration data. The combination of four-wheel independent steering and DRL improves operational flexibility and enables precise trajectory compliance under different operating conditions.</div>

Autonomous exploration and mapping of unknown terrains employing single or multiple robots is an essential task in mobile robotics and has therefore been widely investigated. Nevertheless, given the lack of unified data sets, metrics, and platforms to evaluate the exploration approaches, we develop an autonomous robot exploration benchmark en-titled Explore-Bench. The benchmark involves various explo-ration scenarios and presents two types of quantitative metrics to evaluate exploration efficiency and multi-robot cooperation. Explore-Bench is extremely useful as, recently, deep rein-forcement learning (DRL) has been widely used for robot exploration tasks and achieved promising results. However, training DRL-based approaches requires large data sets, and additionally, current benchmarks rely on realistic simulators with a slow simulation speed, which is not appropriate for training exploration strategies. Hence, to support efficient DRL training and comprehensive evaluation, the suggested Explore-Bench designs a 3-level platform with a unified data flow and 12 × speed-up that includes a grid-based simulator for fast evaluation and efficient training, a realistic Gazebo simulator, and a remotely accessible robot testbed for high-accuracy tests in physical environments. The practicality of the proposed benchmark is highlighted with the application of one DRL-based and three frontier-based exploration approaches. Fur-thermore, we analyze the performance differences and provide some insights about the selection and design of exploration methods. Our benchmark is available at https://github.com/efc-robot/Explore-Bench.

This paper presents a new learning framework that leverages the knowledge from imitation learning, deep reinforcement learning, and control theories to achieve human-style locomotion that is natural, dynamic, and robust for humanoids. We proposed novel approaches to introduce human bias, i.e. motion capture data and a special Multi-Expert network structure. We used the Multi-Expert network structure to smoothly blend behavioral features, and used the augmented reward design for the task and imitation rewards. Our reward design is composable, tunable, and explainable by using fundamental concepts from conventional humanoid control. We rigorously validated and benchmarked the learning framework which consistently produced robust locomotion behaviors in various test scenarios. Further, we demonstrated the capability of learning robust and versatile policies in the presence of disturbances, such as terrain irregularities and external pushes.

This paper proposes a hierarchical framework integrating deep reinforcement learning (DRL) and rule-based methods for multi-lane autonomous driving. We define an instantaneous desired speed (IDS) to mimic the common motivation for higher speed in different traffic situations as an intermediate action. High-level DRL is utilized to generate IDS directly, while the low-level rule-based policies including car following (CF) models and three-stage lane changing (LC) models are governed by the common goal of IDS. Not only the coupling between CF and LC behaviors is captured by the hierarchy, but also utilizing the benefits from both DRL and rule-based methods like more interpretability and learning ability. Owing to the decomposition and combination with rule-based models, traffic flow operations can be taken into account for individually controlled automated vehicles (AVs) with an extension of traffic flow adaptive (TFA) strategy through exposed critical parameters. A comprehensive evaluation for the proposed framework is conducted from both the individual and system perspective, comparing with a pure DRL model and widely used rule-based model IDM with MOBIL. The simulation results prove the effectiveness of the proposed framework.

To fulfill high-level automation, an automated vehicle needs to learn to make decisions and control its movement under complex scenarios. Due to the uncertainty and complexity of the driving environment, most classical rule-based methods cannot solve the problem of complicated decision tasks. Deep reinforcement learning has demonstrated impressive achievements in many fields such as playing games and robotics. However, a direct application of reinforcement learning algorithm for automated driving still face challenges in handling complex driving tasks. In this paper, we proposed a hierarchical reinforcement learning based architecture for decision making and control of lane changing situations. We divided the decision and control process into two correlated processes: 1) when to conduct lane change maneuver and 2) how to conduct the maneuver. To be specific, we first apply Deep Q-network (DQN) to decide when to conduct the maneuver based on the consideration of safety. Subsequently, we design a Deep Q-learning framework with quadratic approximator for deciding how to complete the maneuver in longitudinal direction (e.g. adjust to the selected gap or just follow the preceding vehicle). Finally, a polynomial lane change trajectory is generated and Pure Pursuit Control is implemented for path tracking for the lane change situation. We demonstrate the effectiveness of this framework in simulation, from both the decision-making and control layers.

Music emotion recognition and generation play a vital role in creating emotionally resonant experiences in human-computer interaction. Deep reinforcement learning (DRL) offers promising capabilities for dynamically modeling and generating music aligned with human emotional states. However, existing methods often struggle with rigid emotional mapping, a lack of real-time adaptability, and limited generalization across diverse musical contexts. To address these limitations, this study proposes an Emotion-Conditioned Deep Reinforcement Learning (EC-DRL) framework that integrates emotion-aware representations into the reward mechanism of a DRL agent. The system employs deep neural networks to extract high-level audio features and map them onto valence-arousal emotion space, guiding the music generation process through a reinforcement learning policy optimized for emotional congruence. This framework is applied in adaptive soundtrack generation for interactive applications such as video games, where music responds dynamically to emotional cues derived from gameplay and user interactions. Experimental results demonstrate that the EC-DRL framework significantly improves emotional accuracy, coherence, and user satisfaction compared to traditional sequence-based generation models. The proposed system shows the potential for creating emotionally intelligent music systems that can adapt in real-time and produce expressive musical output. EC-DRL achieves the mapping accuracy by 98%, emotional congruence score by 0.9%, real-time responsiveness by 280 ms, reward function optimization by 9.5%, audio feature extraction quality by 86%, policy convergence rate by 0.8%, user satisfaction score by 8.9%, and cross-domain generalization by 88%.

In this work, we propose a novel data-driven approach to recover missing or corrupted motion capture data, either in the form of 3D skeleton joints or 3D marker trajectories. We construct a knowledge-base that contains prior existing knowledge, which helps us to make it possible to infer missing or corrupted information of the motion capture data. We then build a kd-tree in parallel fashion on the GPU for fast search and retrieval of this already available knowledge in the form of nearest neighbors from the knowledge-base efficiently. We exploit the concept of histograms to organize the data and use an off-the-shelf radix sort algorithm to sort the keys within a single processor of GPU. We query the motion missing joints or markers, and as a result, we fetch a fixed number of nearest neighbors for the given input query motion. We employ an objective function with multiple error terms that substantially recover 3D joints or marker trajectories in parallel on the GPU. We perform comprehensive experiments to evaluate our approach quantitatively and qualitatively on publicly available motion capture datasets, namely CMU and HDM05. From the results, it is observed that the recovery of boxing, jumptwist, run, martial arts, salsa, and acrobatic motion sequences works best, while the recovery of motion sequences of kicking and jumping results in slightly larger errors. However, on average, our approach executes outstanding results. Generally, our approach outperforms all the competing state-of-the-art methods in the most test cases with different action sequences and executes reliable results with minimal errors and without any user interaction.

In the domain of continuous control, deep reinforcement learning (DRL) demonstrates promising results. However, the dependence of DRL on deep neural networks (DNNs) results in the demand for extensive data and increased computational cost. To address this issue, a novel hybrid actor-critic reinforcement learning (RL) framework is introduced. The proposed framework integrates the broad learning system (BLS) with DNN, aiming to merge the strengths of both distinct architectural paradigms. Specifically, the critic network employs BLS for rapid value estimation via ridge regression, while the actor network retains the DNN structure to optimize policy gradients. This hybrid design is generalizable and can enhance existing actor-critic algorithms. To demonstrate its versatility, the proposed framework is integrated into three widely used actor-critic algorithms-deep deterministic policy gradient (DDPG), soft actor-critic (SAC), and twin delayed DDPG (TD3), resulting in BLS-augmented variants. The experimental results reveal that all BLS-enhanced versions surpass their original counterparts in terms of training efficiency and accuracy. These improvements highlight the suitability of the proposed framework for real-time control scenarios, where computational efficiency and rapid adaptation are critical.

In this paper, the downlink packet scheduling problem for cellular networks is modeled, which jointly optimizes throughput, fairness and packet drop rate. Two genie-aided heuristic search methods are employed to explore the solution space. A deep reinforcement learning (DRL) framework with A2C algorithm is proposed for the optimization problem. Several methods have been utilized in the framework to improve the sampling and training efficiency and to adapt the algorithm to a specific scheduling problem. Numerical results show that DRL outperforms the baseline algorithm and achieves similar performance as genie-aided methods without using the future information.

This paper proposes an intelligent game confrontation model for a wargame based on imitation learning and deep reinforcement learning, given light of the supremacy of reinforcement learning for training in complex environments. In the context of simulating the battle between the red team and the blue team, a pre-training model based on expert empirical data is produced using imitation learning, and a TD3 algorithm based on the attention mechanism is further designed to build an experience pool using priority experience replay. Finally, the model is enhanced using the self-play approach to increase its training efficiency. Experiments conducted after training demonstrate that the model has a superior training impact, and the winning rate in simulation training is enhanced by 8% compared to the original model.