Omnidirectional-Wheel Conveyor Path Planning and Sorting Using Reinforcement Learning Algorithms

According to the problem of obstacles avoidance path planning in mobile robot system, this article uses the Q-learning algorithm in reinforcement learning to solve it. The original Q-learning algorithm has the problem of low learning efficiency, then an improved algorithm is proposed that adds a layer of learning process based on it. So that the mobile robot can find obstacles and target positions as soon as possible, which accelerates the efficiency of path planning and improves the efficiency. Finally, the path planning is established by a grid method on Python, and the comparison between the original algorithm, the improved algorithm proves that the learning efficiency of the algorithm is improved.

This paper presents a survey of the most recent literature developments in the field of computational guidance and control for space applications. In recent years, the space sector has changed significantly, shifting towards solutions requiring increased performance and autonomy. This change has also been accelerated by the technical and economic needs of the new space approach. However, traditional guidance and control techniques cannot always meet the increasing requirements of autonomy and performance. For this reason, computational guidance and control has become of increasing interest to the aerospace sector. In contrast to traditional guidance and control, computational guidance and control generates commands in real time by using on-board numerical computation. It represents, therefore, a fundamental step on the way to system autonomy and autonomous operations, with the potential to increase system capabilities and reduce operational costs. On-board computation introduces, however, a series of challenges, including the need to develop algorithms which are computationally efficient, reliable and robust. The literature has classified computational guidance and control techniques into two major categories, namely model-based and data-based. The model-based category encompasses all techniques in which the guidance and control laws are explicitly related to the system dynamic and kinematic models. On the contrary, data-based techniques infer guidance and control laws from a set of data, derived from the models, but do not explicitly use such models in the algorithm itself. Model-based guidance and control includes optimal control, model predictive control and convex optimisation. For the specific case of space attitude re-orientation manoeuvres, additional techniques are path planning and artificial potential functions. The general concept behind the Model Predictive Control (MPC) framework is to solve an optimal control problem repeatedly, for example whenever new measurements are available. The MPC implements only the first element of the returned solution, and the process is then repeated. By doing so, the system is better able to cope autonomously with model uncertainties, failures, or errors. For spacecraft with limited on-board capability, having to repeatedly solve an optimisation problem on-board could represent a challenge. In this case, explicit implementations of the MPC have been proposed, in which the guidance and control laws are precomputed offline and implemented on-board using look-up tables or simple functions. If the optimal control problem of the MPC is convex, or if it can be convexified, convex optimisation can be used directly. If the problem cannot be convexified, the solution can be found by sequentially solving a convex approximation of the original non-convex problem. Convex optimisation can be used either repeatedly in the context of an MPC framework, or as a single function call to generate guidance and control laws, e.g. at the beginning of an attitude re-orientation manoeuvre. Convex optimisation has received increased attention in recent years, also due to its suitability to on-board implementation. In fact, if the optimal control problem is convex, it can be solved reliably and efficiently on-board. Convex optimisation has been successfully applied to the space sector with the notable examples of the SpaceX rocket booster landing and of the Mars pinpoint landing demonstration aboard the Masten Xombie sounding rocket. Model-based guidance techniques, not based on optimisation, have been increasingly explored in recent years, as guidance strategies employed in other engineering fields, such as robotics, are being applied to spacecraft. Among these strategies, of particular importance are path planning and artificial potential field methods, due to their applicability to spacecraft autonomous attitude manoeuvres. Path planning algorithms have been applied for decades to ground vehicles and robotic manipulators. These algorithms consist of techniques to find the best route between two points of a graph representing the search space of a problem. For spacecraft re-orientation applications, the literature has proposed different attitude representation conventions and various cost functions associated to the graph search algorithm. Each approach comes with advantages and disadvantages, which will be analysed in the paper. Along with path planning, Artificial Potential Function (APF) techniques have been developed in the field of robotics, and are now also considered for space applications, to track or maintain a specific reference under a set of constraints. The main idea behind these methods is to synthetise the controller using a potential function. This function is specifically built to grant asymptotic stability to the system, according to Lyapunov’s second stability theorem. This formulation grants flexibility to accommodate system constraints, in a computationally light-weight and efficient manner. Data-based guidance and control laws are mainly based on Artificial Intelligence (AI) technologies, and generate guidance commands after training an algorithm with a large set of data. The interest of the space sector towards AI has accelerated in the last years due to the greater maturity of the technology. Some of the most powerful examples lie in the area of machine learning, like reinforcement learning, supervised learning and deep learning. These algorithms are used to extract patterns from large amounts of data derived from complex nonlinear dynamical and kinematical models. Examples of such data can be numerical solutions of an optimal control problem. The patterns extracted from the data allow the system to identify, in a short computational time, the critical parameters needed to relate the state and control variables. Creating a sufficiently large dataset for training the algorithm requires a considerable computational effort, though, and for certain methods optimality cannot be guaranteed due to the lack of theoretical support derived from the optimality conditions. Advancements made in the last years span several research directions and open a wide variety of opportunities and challenges, which need to be examined to guide further exploration. In this paper, both model-based and data-based techniques and applications are presented. The considered techniques include model predictive control, convex optimisation and machine learning. In addition, for the implementation of autonomous attitude manoeuvres, path planning methods and artificial potential function methods are analysed. These techniques have been applied, in the literature, to a range of different applications. The applications discussed in this paper include, in addition to attitude manoeuvres, powered descent landing guidance, low-thrust orbital transfers, proximity operations and station keeping.

The problem of path exploration and planning of agents in unknown environments is a popular application problem in the field of reinforcement learning. In this paper, we propose an improved reinforcement learning algorithm called the Q-Learning algorithm for adaptive exploration based on simulated annealing (AE-SAQL). We apply the algorithm to the agent path planning problem, improve the setting of the reward function and add the feedback information of the environment. By simulating the Metropolis criterion in the annealing algorithm and adding an adaptive adjustment mechanism, the agent fully explores the environment and makes full use of the environmental information, solving the exploration-utilization dilemma during the algorithm and finally enabling the agent to reach the target location safely. Compared with the standard Q-Learning algorithm and SARSA algorithm, AE-SAQL achieves better.

With the accelerated development of smart cities, the concept of a “smart industrial park” in which unmanned ground vehicles (UGVs) have wide application has entered the industrial field of vision. When faced with multiple tasks and heterogeneous tasks, the task execution efficiency of a single UGV is inefficient, thus the task planning research under multi-UGV cooperation has become more urgent. In this paper, under the anti-collision cooperation mechanism for multi-UGV path planning, an improved algorithm with optimized-weighted-speedy Q-learning (OWS Q-learning) is proposed. The slow convergence speed of the Q-learning algorithm is overcome to a certain extent by changing the update mode of the Q function. By improving the selection mode of learning rate and the selection strategy of action, the relationship between exploration and utilization is balanced, and the learning efficiency of multi-agent in complex environments is improved. The simulation experiments in static environment show that the designed anti-collision coordination mechanism effectively solves the coordination problem of multiple UGVs in the same scenario. In the same experimental scenario, compared with the Q-learning algorithm and other reinforcement learning algorithms, only the OWS Q-learning algorithm achieves the convergence effect, and the OWS Q-learning algorithm has the shortest collision-free path for UGVS and the least time to complete the planning. Compared with the Q-learning algorithm, the calculation time of the OWS Q-learning algorithm in the three experimental scenarios is improved by 53.93%, 67.21%, and 53.53%, respectively. This effectively improves the intelligent development of UGV in smart parks.

This paper deals with a new approach based on Q-learning for solving the problem of mobile robot path planning in complex unknown static environments. As a computational approach to learning through interaction with the environment, reinforcement learning algorithms have been widely used for intelligent robot control, especially in the field of autonomous mobile robots. However, the learning process is slow and cumbersome. For practical applications, rapid rates of convergence are required. Aiming at the problem of slow convergence and long learning time for Q-learning based mobile robot path planning, a state-chain sequential feedback Q-learning algorithm is proposed for quickly searching for the optimal path of mobile robots in complex unknown static environments. The state chain is built during the searching process. After one action is chosen and the reward is received, the Q-values of the state-action pairs on the previously built state chain are sequentially updated with one-step Q-learning. With the increasing number of Q-values updated after one action, the number of actual steps for convergence decreases and thus, the learning time decreases, where a step is a state transition. Extensive simulations validate the efficiency of the newly proposed approach for mobile robot path planning in complex environments. The results show that the new approach has a high convergence speed and that the robot can find the collision-free optimal path in complex unknown static environments with much shorter time, compared with the one-step Q-learning algorithm and the Q(λ)-learning algorithm.

In response to the problem of low efficiency and weak obstacle avoidance ability in finding the optimal or suboptimal path for 3D path planning of drones in complex dynamic environments, this paper uses Q-learning algorithm to complete the 3D path planning of drones, aiming to improve their path planning and obstacle avoidance capabilities. Firstly, by constructing a three-dimensional gridded environment model, the system calculates the reward for each state under the influence of natural environment and obstacles, and then guides the drone to avoid obstacles and find the optimal path. The system uses the ε - greedy strategy for exploration and learning, optimizing decisions by continuously updating the Q-table value table. The experimental results show that the drone has a success rate of 93.3% in obstacle avoidance in complex and multi obstacle scenes. Moreover, in terms of average path length, the Q-learning algorithm has shortened it by approximately 20.00%, 11.45%, and 40.39% compared to ant colony algorithm, A* algorithm, and RRT algorithm, respectively. In dynamic wind speed environments, the Q-learning algorithm reduces the path length by about 4% to 11% compared to other algorithms, further demonstrating its effectiveness and advantages in complex environments.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

The naval battlefield is one of the main positions for future conflicts between major powers. The powerful naval battlefield target search capability is the last barrier to carry out maritime training and operations. At the same time, because of its complex and changeable environment and important strategic position, it has become the most difficult battlefield in joint search and rescue. the most central part. In order to reduce the time of target search in maritime battlefield, a real-time path planning method in maritime battlefield based on deep reinforcement learning is proposed. The method proposed in this paper has advantages for path planning in multi-machine collaborative and can meet the requirements of real-time performance. First, a mathematical planning model for target search in the naval battlefield is constructed and mapped into a reinforcement learning model; then, based on the Rainbow deep reinforcement learning algorithm, the state vector, neural network structure and algorithm framework for target search path planning in the naval battlefield are designed with process. Finally, the feasibility and effectiveness of the proposed method are verified by experiments. The sequential exchange strategy and sequential insertion strategy of the A* algorithm are added to improve the optimal path obtained by the algorithm in each iteration. The improved algorithm is applied to the target path planning problem with time window. Through experiments on the Solomon data set, the optimal solution of the path before and after the improved algorithm is compared, and it is proved that the improved algorithm has better performance. In this way, the cooperative behavior of multi-agents can be realized, and the stability of reinforcement learning can be improved. The simulation results show that the length of the path searched by the proposed algorithm and the inflection point of the path are shortened by 19% and 58% respectively compared with the benchmark algorithm, which verifies the superiority of the proposed algorithm.

In this article, we have simulated an agent-based model to investigate the bidding behavior of generation companies (GenCos) in the electricity market under different pricing rules. In a real electricity market, they do not have complete information about the behavior of competitors and therefore make their own decisions based on information that exists on the market-clearing price (MCP) from the past. So considering of high ability of reinforcement learning (RL) algorithms for making decisions on issues with incomplete information, the behavior of companies has been simulated with RL algorithms. In addition, to remove the maximization bias in usual algorithms, we have used the double learning technique for unbiased estimation. The results have shown that companies using Double Q-learning and Double SARSA algorithms, can gain optimal values with unbiased estimation. Furthermore, we investigated market-clearing mechanisms with different pricing rules under an equal rationing policy, in terms of competitiveness in the market and total profit of GenCos. The results showed that in terms of competitiveness, the uniform pricing rule causes more competitive bid prices. However, from the side of the total profit of GenCos, it was seen that with changing from the uniform pricing to the pay-as-bid (P AB), the total profit of companies would reduce.

P2P power trading based on reinforcement learning for nanogrid clusters

We present DDPG-AACO, a hierarchical method for welding manipulator path planning of complex components welding tasks that combines reinforcement learning (RL) with the intelligent optimisation algorithm. The RL agent, trained with the deep deterministic policy gradient (DDPG), learns local path planning policies that control the welding manipulator to safely move between two welding seam endpoints. Next, on a distance matrix constructed by the lengths of local paths between every two welding seam endpoints, the adaptive ant colony optimisation (AACO) algorithm with artificially changed value of pheromones is adopted to realise the global path planning that the welding manipulator traverses all welding seams under a welding direction constraint and has the shortest path length. The simulation results show the effectiveness of the method. The DDPG is better than the deep Q-learning-based methods when performing local path planning. Moreover, the length of the global path with direction constraint can converge to the minimum.

With the wide application of mobile robots in industry, path planning has always been a difficult problem for mobile robots. Reinforcement learning algorithms such as Q-learning play a huge role in path planning. Traditional Q-learning algorithm mainly uses - greedy search policy. But for a fixed search factor -greedy. For example, the problems of slow convergence speed, time-consuming and many continuous action transformations (such as the number of turns during robot movement) are not conducive to the stability requirements of mobile robots in industrial transportation. Especially for the transportation of dangerous chemicals, continuous transformation of turns will increase the risk of objects toppling. This paper proposes a new method based on - greedy 's improved dynamic search strategy is used to improve the stability of mobile robots in motion planning. The experiment shows that the dynamic search strategy converges faster, consumes less time, has less continuous transformation times of action, and has higher motion stability in the test environment.

Path planning is essential for autonomous underwater vehicles (AUVs) to perform tasks. Many existing single-objective path planning methods rely on prior knowledge of the underwater environment. However, extracting prior knowledge is challenging because the underwater environment is influenced by ocean current, complex terrain, and other factors. This paper proposes a two-stage reinforcement learning (RL) algorithm based on trajectory exploration and sequence modeling, called the Soft Actor Critic and Online Decision Transformer (SAC-ODT), which operates without prior knowledge. This algorithm utilizes the SAC strategy exploration capability to generate training data for ODT and uses the ODT strategy optimization capability to plan a smooth, energy-efficient, and safe path. In the first stage, a Multi-Reward Strategy Embedding (MRSE) method is designed to facilitate trajectory exploration with multiple strategies, enabling further training of an ODT with comprehensive decision-making capability. During the second stage, a Condition Prioritized Buffer Update and Sampling Strategy (CP-BUSS) is proposed to enhance the sensitivity of ODT to the reward function, enabling adaptation to various tasks while accelerating high-quality path learning. Experimental results demonstrate that compared to existing RL-based benchmarks, SAC-ODT reduces the path time and energy consumption by 2.7% and 2.5%, respectively, while improving path smoothness by 91.96%.

This paper proposes a reinforcement learning (RL) algorithm for the security problem of state estimation of cyber-physical system (CPS) under denial-of-service (DoS) attacks. The security of CPS will inevitably decline when faced with malicious cyber attacks. In order to analyze the impact of cyber attacks on CPS performance, a Kalman filter, as an adaptive state estimation technology, is combined with an RL method to evaluate the issue of system security, where estimation performance is adopted as an evaluation criterion. Then, the transition of estimation error covariance under a DoS attack is described as a Markov decision process, and the RL algorithm could be applied to resolve the optimal countermeasures. Meanwhile, the interactive combat between defender and attacker could be regarded as a two-player zero-sum game, where the Nash equilibrium policy exists but needs to be solved. Considering the energy constraints, the action selection of both sides will be restricted by setting certain cost functions. The proposed RL approach is designed from three different perspectives, including the defender, the attacker and the interactive game of two opposite sides. In addition, the framework of Q-learning and state–action–reward–state–action (SARSA) methods are investigated separately in this paper to analyze the influence of different RL algorithms. The results show that both algorithms obtain the corresponding optimal policy and the Nash equilibrium policy of the zero-sum interactive game. Through comparative analysis of two algorithms, it is verified that the differences between Q-Learning and SARSA could be applied effectively into the secure state estimation in CPS.

Fast road emergency response can minimize the losses caused by traffic accidents. However, emergency rescue on urban arterial roads is faced with the high probability of congestion caused by accidents, which makes the planning of rescue path complicated. This paper proposes a refined path planning method for emergency rescue vehicles on congested urban arterial roads during traffic accidents. Firstly, a rescue path planning environment for emergency vehicles on congested urban arterial roads based on the Markov decision process is established, which focuses on the architecture of arterial roads, taking the traffic efficiency and vehicle queue length into consideration of path planning; then, the prioritized experience replay deep Q-network (PERDQN) reinforcement learning algorithm is used for path planning under different traffic control schemes. The proposed method is tested on the section of East Youyi Road in Xi’an, Shaanxi Province, China. The results show that compared with the traditional shortest path method, the rescue route planned by PERDQN reduces the arrival time to the accident site by 67.1%, and the queue length at upstream of the accident point is shortened by 16.3%, which shows that the proposed method is capable to plan the rescue path for emergency vehicles in urban arterial roads with congestion, shorten the arrival time, and reduce the vehicle queue length caused by accidents.