A Hierarchical Framework for Quadruped Omnidirectional Locomotion Based on Reinforcement Learning

The ability to maintain balance and stability is a major challenge for quadruped robots in locomotion. For example, when walking in the unstructured environment, the robot not only needs to adjust its gait according to the terrain information, but also needs to tune the body posture based on the real-time body posture acquired from inertial measurement unit (IMU) to maintain stability. In this paper, we present a method that focuses on the design of the posture correction controller based on the IMU feedback for adaptive slope walking for a quadruped robot. To this end, the posture correction controller is integrated into the foot trajectory generator to maintain balance. We experimentally validated our approach both on simulation and real quadruped robot. Compared with the one only with the foot trajectory generator, the quadruped robot can walk more stable with the posture correction controller integrated into the foot trajectory generator and has a better locomotion ability. Our method increases locomotion stability when walking on the slopes.

In reinforcement learning, the reward function has a significant impact on the performance of the agent. However, determining the appropriate value of this reward function requires many attempts and trials. Although many automated reinforcement learning methods have been proposed to find an appropriate reward function, their proof is lacking in complex environments such as quadrupedal locomotion. In this paper, we propose a method to automatically tune the scale of the dominant reward functions in reinforcement learning of a quadrupedal robot. Reinforcement learning of the quadruped robot is very sensitive to the reward function, and recent outstanding research results have put a lot of effort into reward shaping. In this paper, we propose an automated reward shaping method that automatically adjusts the reward function scale appropriately. We select some dominant reward functions, arrange their weights in a certain unit, and then calculate their gait scores so that we can select the agent with the highest score. This gait score was defined to reflect the stable walking of the quadrupedal robot. Additionally, quadrupedal locomotion learning requires reward functions of different scales depending on the robot’s size and shape. Therefore, we evaluate the performance of the proposed method on two different robots.

We present a unified control framework that generates dynamic walking motions for biped and quadruped robots with online relative footstep optimization. The footstep optimization is formulated as a discrete-time Model Predictive Control problem which determines future footstep locations. The framework has a hierarchical structure consisting of three layers: footstep planner, trajectory generator and whole-body controller. The footstep planner plans next footstep position based on Linear Inverted Pendulum (LIP) model. Relative footstep optimization is proposed to enable automatic footstep planning without the use of any predefined footstep sequences. The trajectory generator will generate CoM and feet trajectory given the next footstep placement. In order to generalize to quadruped robots, "virtual leg" concept has been used to coordinate leg pair movement. The whole-body inverse dynamic controller calculates joint torques to track given Cartesian reference trajectories. To include under-actuation into consideration, contact vertices formulation of ground reaction forces (GRFs) has been adopted. Generalized whole-body controller can handle biped robot with line feet as well as quadruped robots with point feet walking with dynamic gaits. Several simulations have been performed to demonstrate the robustness and generality of the proposed framework.

Due to the strongly environmental adaptation, quadruped robot becomes a research hot spot. The single leg control lays foundation for the control of quadruped robot. The operational space control which is used in the quadruped robot single leg control algorithms usually strongly depends on the accuracy of the model. This paper applies RBF (radial basis function) neural network adaptive control on the single leg, and a kind of control parameters adjustment method which is based on reinforcement learning is proposed. The result shows the algorithm effectively improves the control accuracy and convergence speed under the high-dynamic condition.

It often takes a lot of trial and error to get a quadruped robot to learn a proper and natural gait directly through reinforcement learning. Moreover, it requires plenty of attempts and clever reward settings to learn appropriate locomotion. However, the success rate of network convergence is still relatively low. In this paper, the referred trajectory, inverse kinematics, and transformation loss are integrated into the training process of reinforcement learning as prior knowledge. Therefore reinforcement learning only needs to search for the optimal solution around the referred trajectory, making it easier to find the appropriate locomotion and guarantee convergence. When testing, a PD controller is fused into the trained model to reduce the velocity following error. Based on the above ideas, we propose two control framework - single closed-loop and double closed-loop. And their effectiveness is proved through experiments. It can efficiently help quadruped robots learn appropriate gait and realize smooth and omnidirectional locomotion, which all learned in one model.

Quadruped robots have great agility, flexibility and stability, which enables them to walk through uneven terrain. Motion control of legged robots is always a difficult problem. Previous approaches mostly either use a predefined gait that results in clumsy and unnatural behavior, or use reinforcement learning approach to generate a gait strategy that needs longtime computation and elegant network design. In this paper, we present an effective approach that uses deep reinforcement learning with prior knowledge to optimize the gait of quadruped robot. By using a specific quadruped robot walking gait as a priori knowledge, this approach adopts the technique of distributed proximal policy optimization to optimize the search for better gait. The proposed approach does not require modelling of complex robots, and has good network convergence speed and learning effect. Simulation results demonstrate that our proposed approach converges faster than other deep reinforcement learning methods without prior knowledge. Besides, our achieved gait has higher speed that is 50% faster than the trot gait without optimization.

Dynamic locomotion of legged robots is a critical yet challenging topic in expanding the operational range of mobile robots. It requires precise planning when possible footholds are sparse, robustness against uncertainties and disturbances, and generalizability across diverse terrains. Although traditional model-based controllers excel at planning on complex terrains, they struggle with real-world uncertainties. Learning-based controllers offer robustness to such uncertainties but often lack precision on terrains with sparse steppable areas. Hybrid methods achieve enhanced robustness on sparse terrains by combining both methods but are computationally demanding and constrained by the inherent limitations of model-based planners. To achieve generalized legged locomotion on diverse terrains while preserving the robustness of learning-based controllers, this paper proposes an attention-based map encoding conditioned on robot proprioception, which is trained as part of the controller using reinforcement learning. We show that the network learns to focus on steppable areas for future footholds when the robot dynamically navigates diverse and challenging terrains. We synthesized behaviors that exhibited robustness against uncertainties while enabling precise and agile traversal of sparse terrains. In addition, our method offers a way to interpret the topographical perception of a neural network. We have trained two controllers for a 12-degrees-of-freedom quadrupedal robot and a 23-degrees-of-freedom humanoid robot and tested the resulting controllers in the real world under various challenging indoor and outdoor scenarios, including ones unseen during training.

Safety is a critical component of autonomous systems and remains a challenge for learning-based policies to be utilized in the real world. In particular, policies learned using reinforcement learning often fail to generalize to novel environments due to unsafe behavior. In this paper, we propose Sim-to-Lab-to-Real to bridge the reality gap with a probabilistically guaranteed safety-aware policy distribution. To improve safety, we apply a dual policy setup where a performance policy is trained using the cumulative task reward and a backup (safety) policy is trained by solving the Safety Bellman Equation based on Hamilton-Jacobi (HJ) reachability analysis. In Sim-to-Lab transfer, we apply a supervisory control scheme to shield unsafe actions during exploration; in Lab-to-Real transfer, we leverage the Probably Approximately Correct (PAC)-Bayes framework to provide lower bounds on the expected performance and safety of policies in unseen environments. Additionally, inheriting from the HJ reachability analysis, the bound accounts for the expectation over the worst-case safety in each environment. We empirically study the proposed framework for ego-vision navigation in two types of indoor environments with varying degrees of photorealism. We also demonstrate strong generalization performance through hardware experiments in real indoor spaces with a quadrupedal robot. See https://sites.google.com/princeton.edu/sim-to-lab-to-real for supplementary material.

To address the challenges of real-time decision-making and resource optimization in multi-agent cooperative interception tasks within dynamic environments, this paper proposes a hierarchical framework for reinforcement learning-based interception algorithm (HFRL-IA). By constructing a hierarchical Markov decision process (MDP) model based on dynamic game equilibrium theory, the complex interception task is decomposed into two hierarchically optimized stages: dynamic task allocation and distributed path planning. At the high level, a sequence-to-sequence reinforcement learning approach is employed to achieve dynamic bipartite graph matching, leveraging a graph neural network encoder–decoder architecture to handle dynamically expanding threat targets. At the low level, an improved prioritized experience replay multi-agent deep deterministic policy gradient algorithm (PER-MADDPG) is designed, integrating curriculum learning and prioritized experience replay mechanisms to effectively enhance the interception success rate against complex maneuvering targets. Extensive simulations in diverse scenarios and comparisons with conventional task assignment strategies demonstrate the superiority of the proposed algorithm. Taking a typical scenario of 10 agents intercepting as an example, the HFRL-IA algorithm achieves a 22.51% increase in training rewards compared to the traditional end-to-end MADDPG algorithm, and the interception success rate is improved by 26.37%. This study provides a new methodological framework for distributed cooperative decision-making in dynamic adversarial environments, with significant application potential in areas such as maritime multi-agent security defense and marine environment monitoring.

Automatic decision-making approaches, such as reinforcement learning (RL), have been applied to (partially) solve the resource allocation problem adaptively in the cloud computing system. However, a complete cloud resource allocation framework exhibits high dimensions in state and action spaces, which prohibit the usefulness of traditional RL techniques. In addition, high power consumption has become one of the critical concerns in design and control of cloud computing systems, which degrades system reliability and increases cooling cost. An effective dynamic power management (DPM) policy should minimize power consumption while maintaining performance degradation within an acceptable level. Thus, a joint virtual machine (VM) resource allocation and power management framework is critical to the overall cloud computing system. Moreover, novel solution framework is necessary to address the even higher dimensions in state and action spaces. In this paper, we propose a novel hierarchical framework for solving the overall resource allocation and power management problem in cloud computing systems. The proposed hierarchical framework comprises a global tier for VM resource allocation to the servers and a local tier for distributed power management of local servers. The emerging deep reinforcement learning (DRL) technique, which can deal with complicated control problems with large state space, is adopted to solve the global tier problem. Furthermore, an autoencoder and a novel weight sharing structure are adopted to handle the high-dimensional state space and accelerate the convergence speed. On the other hand, the local tier of distributed server power managements comprises an LSTM based workload predictor and a model-free RL based power manager, operating in a distributed manner.

In this paper, we propose a quadruped robot with asymmetric bionic legs and a bionic spine with one degree of freedom (DOF). The robot’s forelimb has a joint configuration of 1T (translational joint) + 3R (rotational joint), while the hindlimb has a joint configuration of 4R. This paper provides a bionic mechanism design and verification method that applies reinforcement learning to the controller of the newly designed quadruped robot, which addresses the increasing complexity and number of joints. The control strategy presented in the paper is utilized to adapt the robot to rough and sloped terrain in both the simulated environment and the actual world. On the one hand, this study illustrates the adaptability of the new configuration robot to rough and sloped terrains and, on the other, that the suggested approach can efficiently verify the new bionic mechanism and speed up the process of mechanism design iterations.KeywordsQuadruped robotBionic configuration designReinforcement learningMotion control

Recent evidence in neuroscience and psychology suggests that a single reinforcement learning (RL) algorithm only accounts for less than 60% of the variance of human choice behavior in an uncertain and dynamic environment, where the amount of uncertainty in state-action-state transitions drift over time. The prediction performance further decreases when the size of the state space increases. We proposed a hierarchical context-dependent RL control framework that dynamically exerted control weights on model-based (MB) and multiple model-free (MF) RL strategies associated with different task goals. To properly assess the validity of the proposed method, we considered a two-stage Markov decision task (MDT) in which the three different types of context changed over time. We trained 57 different RL control models on a Caltech MDT data set; then, we assessed their prediction performance using a Bayesian model comparison. This large-scale computer simulation analysis revealed that the model providing the most accurate prediction was the version that implemented the competition between the MB and multiple goal-dependent MF RL strategies. The present study demonstrates the applicability of the goal-driven RL control to a variety of real-world human-robot interaction scenarios.

The verification of Unmanned Aerial Systems (UAS) trajectory generation in uncertain environments is a critical challenge, particularly when using reinforcement learning (RL) methods. The unpredictability of RL models, together with the lack of formal verification standards for UAS and AI systems, complicates the safety and reliability of autonomous decision-making. This paper proposes a learning framework as a solution towards improving the verifiability of RL-based trajectory generation. We propose an incremental training process where the RL model is progressively trained in more complex and uncertain scenarios. By controlling the complexity of training environments, the framework can guide the system towards safer, more predictable behaviour, making it easier to verify its performance under varying conditions. This approach also helps reduce the risks associated with RL's non-deterministic nature by creating a better understanding of robustness and stability in the model's decision-making process. Additionally, we discuss the current means of certifying airborne software and the need for new standards and methodologies to ensure that learning-based systems can be rigorously evaluated for safety and operational readiness in dynamic, real-world environments.

Mobile robots and their motion control are two of the challenging and active research areas nowadays. Wheeled robots can be very fast and stable when moving on smooth terrain. On the other hand, legged robots are far superior when it comes to rough terrains and environments that contain many obstacles, which is the case of many real-life situations. In this article, and the design and the implementation of a hexapod robot with 18 DOF is presented. This robot contains six legs attached to a main body known as the trunk. Each leg will have three links known as the coxa, the femur and the tibia, resulting in 3 joints and consequently 3 DOF for each leg powered by 3 DC servo motors. The amount of rotation for each servo motor is controlled by an Arduino microcontroller boards. Inverse and forward kinematics are studied to produce a mathematical model for the motion of the legs based on a predefined trajectory for the end effector of each leg. The robot is designed using CAD software and a simulation is carried out in order to validate the inverse and forward kinematics results. In addition, an experimental prototype setup is built and tested. Experimental results for the robot motion show smooth motion for the robot. It is tested on different types of terrains, namely, smooth and rough terrains, and the results are promising in each case. Video for the motion will be provided.

Discerning Lines of Demarcation is an investigation into the accumulated landscape of distressing times. Situations of mass destruction, loss of family, substance abuse, domestic violence, loss of friendship, and uncertain health have all been encountered within a steady progression in the last five to six years of my life. The digestion of these situations has been slow as the events overlap and intertwine each other. I have tweezed and distilled these circumstances. This is described through different types of terrains that create physical boundaries to represent psychological fears or events. Tied to Your Hate; How Much More Will Fall, Untitled, Loss Can Ruin but Growth Can Be Found, and Precarious Footing are all pieces that reach for a greater understanding of many trying events. The work uses the format of installation as the vehicle in which to articulate the landscapes or yards. This format allows the viewer to be inside the workings of the various pieces.