Research on path tracking control method of unmanned surface vehicle based on deep reinforcement learning

The objective is to evaluate the performance of an explicit model predictive control for controlling the room temperature actuated by the on-off air conditioner to minimize the energy usage by reducing the working time of a compressor. There are a lot of on-off air conditioners already installed in the educational building in Thailand. To replace all of those by the inverter air conditioner or HVAC system, the high investment cost is required. Therefore, the on-off air conditioners continue to work in such building. The effective solution may be the low cost control unit which is simple like a remote control. The method is to design the remote control embedded by the temperature sensor and the explicit model predictive control that is based on the off-line optimization. The experiment is performed in the laboratory room with dimension 4×7×3.5 m (WxLxH) actuated by Saijo Denki 18,320 Btu on-off air conditioner. The experimental results reveal that the temperature reaches the desired value 25 degree Celsius within 10 minute and keeps fluctuating between 23 and 26 degree Celsius. The working time of compressor is reduced 33.33 %. The result shows that the explicit model predictive control is effective for the remote control application in terms of the implementation and energy saving.

This thesis is concerned with different topics in multi-parametric programming and explicit model predictive control, with particular emphasis on hybrid systems. The main goal is to extend the applicability of these concepts to a wider range of problems of practical interest, and to propose algorithmic solutions to challenging problems such as constrained dynamic programming of hybrid linear systems and nonlinear explicit model predictive control. The concepts of multi-parametric programming and explicit model predictive control are presented in detail, and it is shown how the solution to explicit model predictive control may be efficiently computed using a combination of multi-parametric programming and dynamic programming. A novel algorithm for constrained dynamic programming of mixed-integer linear problems is proposed and illustrated with a numerical example that arises in the context of inventory scheduling. Based on the developments on constrained dynamic programming of mixed-integer linear problems, an algorithm for explicit model predictive control of hybrid systems with linear cost function is presented. This method is further extended to the design of robust explicit controllers for hybrid linear systems for the case when uncertainty is present in the model. The final part of the thesis is concerned with developments in nonlinear explicit model predictive control. By using suitable model reduction techniques, the model captures the essential nonlinear dynamics of the system, while the achieved reduction in dimensionality allows the use of nonlinear multi-parametric programming methods.

Design automation is undergoing a new generation of changes caused by artificial intelligence technologies represented by deep learning and reinforcement learning. Notably, the advantages of deep reinforcement learning in addressing solution optimisation and decision-making tasks with cognitive automation functionality have garnered attention in design. In the context of surrogate model-driven engineering design optimisation, this paper addresses current research challenges such as reliance on domain knowledge for local development, shortcomings in the self-learning and adaptive capabilities of optimisation algorithms for global exploration, etc. Centred around the deep reinforcement learning model, Deep Q-learning, and complemented by self-organising maps and neural network technologies, we propose a methodology considering multi-fidelity simulation data for design space exploration. This approach effectively reduces sampling costs and enables the optimisation model to learn the optimal direction for high-precision predictions and achieve rapid, accurate optimisation. Finally, the effectiveness of the proposed method is comprehensively validated through four typical optimisation scenarios and a case study involving the optimisation of a wheeled robot's suspension swing arm structure. This work will be a crucial reference for applying deep reinforcement learning in simulation-driven engineering design optimisation.

Deep reinforcement learning (DRL) is one of the emerging areas of machine learning which focuses on maximized rewards. DRL is a type of machine learning that combines reinforcement learning and deep learning. It uses a series of algorithms to enable an agent to learn how to make decisions in a complex environment. DRL is a subset of artificial intelligence that focuses on making decisions based on the environment and the rewards associated with each action.The goal of DRL is to maximize the long-term reward of an agent. In order to do this, the agent must use a combination of deep learning, reinforcement learning and other AI techniques to learn which actions will lead to the highest reward. DRL is used to solve a variety of problems, from playing video games to controlling robots. It is also used in autonomous driving and robotics, as well as for financial trading. DRL is a powerful tool for solving complex problems and has been used in a variety of research projects. DRL has the potential to revolutionize the way we interact with machines and the environment.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

Path planning refers to that the mobile robot can obtain the surrounding environment information and its own state information through the sensor carried by itself, which can avoid obstacles and move towards the target point. Deep reinforcement learning consists of two parts: reinforcement learning and deep learning, mainly used to deal with perception and decision-making problems, has become an important research branch in the field of artificial intelligence. This paper first introduces the basic knowledge of deep learning and reinforcement learning. Then, the research status of deep reinforcement learning algorithm based on value function and strategy gradient in path planning is described, and the application research of deep reinforcement learning in computer game, video game and autonomous navigation is described. Finally, I made a brief summary and outlook on the algorithms and applications of deep reinforcement learning.

The power grid requires doubly‐fed induction generators (DFIGs) to effectively suppress rotor currents and provide reactive power support during low voltage ride‐through (LVRT) events. However, traditional virtual impedance methods struggle to flexibly coordinate different control objectives under various fault conditions, thereby failing to fully realize their control potential. To address this, this paper proposes a decoupled virtual impedance control strategy. Virtual resistance and inductance are independently designed to suppress zero‐sequence and negative‐sequence overcurrents, achieving effective overcurrent mitigation. Furthermore, explicit model predictive control (E‐MPC) is employed to dynamically adjust the virtual impedance values, enhancing the LVRT capability of DFIGs. Additionally, the E‐MPC cost function incorporates reactive power support to coordinate current suppression and voltage support. Since the control law of E‐MPC is designed offline, the online computation time can be greatly reduced. The proposed strategy is verified by 1.5 MW DFIG model in Matlab/Simulink and experiment. Results show that LVRT performances under diverse fault conditions have been significantly improved.

With the improvement in the intelligence level of UAVs and the development of cluster control technology, the intelligent decision-making method of UAV cluster confrontation will become the key technology of UAV combat in the future. The UAV cluster confrontation learning environment is of complex characteristics of high dimension, non-linearity, incomplete information, continuous action space, and so on. Recently, artificial intelligence technology represented by deep learning and reinforcement learning has made a great breakthrough. Deep reinforcement learning has shown a great ability to solve intelligent decision-making problems in complex environments. In this paper, enlightened by the multi-agent centralized training distributed execution framework and the idea of maximum policy entropy, we propose a deep reinforcement learning method of the multi-agent soft actor-critic (MASAC) with incomplete information. A game model for UAV cluster confrontation based on multi-agent deep reinforcement learning is established, and a continuous space multiple UAV combat environment is constructed. Simulation experiments are performed on the asymmetric confrontation of UAV clusters with red and blue teams. The experimental results show that MASAC outperforms the existing popular multi-agent deep reinforcement learning methods, making the game players converge to a higher return equilibrium point of the game. Moreover, the convergence of MASAC is investigated and analyzed extensively, and the results show that MASAC is of good convergence and stability, ensuring the practicability of MASAC in intelligent decision-making regarding UAV cluster confrontation.

This paper discusses an autonomous driving control strategy based on Deep Reinforcement Learning (DRL), which aims to improve the decision-making ability of autonomous driving system in complex traffic environments. Deep reinforcement learning has a wide range of applications in many fields, such as robotics and medicine. Autonomous driving has emerged as a significant research focus in recent years. By combining deep learning and reinforcement learning, the model is able to autonomously learn and optimize driving behavior under dynamically changing road conditions. The DRL-based control strategy performs well in vehicle obstacle avoidance, pedestrian recognition, and traffic rule compliance in the face of complex environments such as city streets, intersections, and congested road sections, significantly improving the safety and efficiency of autonomous driving. This article will first introduce deep reinforcement learning. Then, the autonomous driving control strategy based on deep reinforcement learning is introduced. This research provides valuable insights for developing and implementing DRL-based autonomous driving systems.

A novel traffic signal split approach based on Explicit Model Predictive Control

Trajectory tracking is a basic function required for autonomous helicopters, but it also poses challenges to control design due to the complexity of helicopter dynamics. This article introduces an explicit model predictive control (MPC) to solve this problem, which inherits the advantages of non-linear MPC but eliminates time-consuming online optimization. The explicit solution to the non-linear MPC problem is derived using Taylor expansion and exploiting the helicopter model. With the explicit MPC solution, the control signals can be calculated instantaneously to respond to the fast dynamics of helicopters and suppress disturbances immediately. On the other hand, the online optimization process can be removed from the MPC framework, which can accelerate the software development and simplify onboard hardware. Due to these advantages of the proposed method, the overall control framework has a low complexity and high reliability, and it is easy to deploy on small-scale helicopters. The proposed explicit non-linear MPC has been successfully validated in simulations and in actual flight tests using a Trex-250 small-scale helicopter.

This work deals with the development of double closed loop control based on explicit model predictive control (MPC) for permanent magnet synchronous motor (PMSM) system with constraints. The aim of the proposed controller is to solve the constrained optimization problem for fast-sampling time PMSM system and improve the robustness of the system. The recreated mathematical model of PMSM is introduced at first. Then the proposed explicit MPC controller is designed based on the reformed model. The explicit controller is employed to control the mechanical speed and the q-axis current, and the proportional-integral (PI) control is employed to control d-axis current. The simulation results demonstrate that the proposed double closed loop control method is effective when the working condition of PMSM changes and the performance of the speed and current is satisfactory.

In recent years, deep reinforcement learning (DRL) has made impressive achievements in many fields. However, existing DRL algorithms usually require a large amount of exploration to obtain a good action policy. In addition, in many complex situations, the reward function cannot be well designed to meet task requirements. These two problems will make it difficult for DRL to learn a good action policy within a relatively short period. The use of expert data can provide effective guidance and avoid unnecessary exploration. This study proposes a deep imitation reinforcement learning (DIRL) algorithm that uses a certain amount of expert demonstration data to speed up the training of DRL. In the proposed method, the learning agent imitates the expert's action policy by learning from demonstration data. After imitation learning, DRL is used to optimise the action policy in a self‐learning way. By experimental comparison on a video game called the Mario racing game, it is shown that the proposed DIRL algorithm with expert demonstration data can obtain much better performance than previous DRL algorithms without expert guidance.

The Lunar Gravity Compensation Facility (LGCF) is a critical component in ground tests for a crewed lunar roving vehicle (CLRV). The track-following servo subsystem’s performance is of critical importance in the LGCF, as it needs to achieve high-precision tracking of the CLRV’s fast, wide range of motion in the horizontal direction. The subsystem must also operate within various constraints, including those related to speed, acceleration, and position. These requirements introduce new challenges to both the design and control of the subsystem. To tackle these challenges, this paper employs a Permanent-Magnet Synchronous Motor (PMSM) vector control method based on Space Vector Pulse Width Modulation (SVPWM) to achieve accurate speed tracking. Additionally, this paper presents an Explicit Model Predictive Control (EMPC) strategy for precise position servo control of the track-following system under multi-parameter constraints. The simulation model of the track-following servo subsystem is established based on the above methods. The simulation results demonstrate that the position tracking error of the gravity compensation system, constructed using the above method combined with EMPC control, is less than 0.2 m. The control performance of the EMPC is significantly better than those of the PI and LQI controllers. The influence of errors on the drawbar pull is within 12.5%, and its effect on the compensation force is negligible. These results provide theoretical support for the design of a track-following servo subsystem.

This paper investigates the vision-based autonomous driving with deep learning and reinforcement learning methods. Different from the end-to-end learning method, our method breaks the vision-based lateral control system down into a perception module and a control module. The perception module which is based on a multi-task learning neural network first takes a driver-view image as its input and predicts the track features. The control module which is based on reinforcement learning then makes a control decision based on these features. In order to improve the data efficiency, we propose visual TORCS (VTORCS), a deep reinforcement learning environment which is based on the open racing car simulator (TORCS). By means of the provided functions, one can train an agent with the input of an image or various physical sensor measurement, or evaluate the perception algorithm on this simulator. The trained reinforcement learning controller outperforms the linear quadratic regulator (LQR) controller and model predictive control (MPC) controller on different tracks. The experiments demonstrate that the perception module shows promising performance and the controller is capable of controlling the vehicle drive well along the track center with visual input.