A hidden Markov model and reinforcement learning‐based strategy for fault‐tolerant control

In a Packed U Cell Multilevel Inverter, by optimizing the fractional order PID controller can improve performance by minimizing rise time, settling time, and steady-state error for capacitor balancing. Therefore, the Animated Oat Optimization Algorithm was used to optimize gain parameters of Fractional-order PID controllers for capacitor balancing. Based on the natural dispersal and hygroscopic motion of oat seeds, the Animated Oat Optimization algorithm provides a powerful global search capability and robust convergence behaviour. Animated Oat Optimization based Fractional order PID controllers are compared to conventional PID, Fractional order PID, Particle Swarm Optimization based Fractional order PID, etc., under identical operating conditions to validate their effectiveness. Some key transient performance measures are analyzed, including rise time, settling time, peak overshoot, steady-state error, and total harmonic distortion of the inverter output voltage. Additionally, control precision and system efficiency are evaluated using standard performance indices such as Integral Square Error, Integral Absolute Error,Integral Time Absolute Error, and Integral Time Square Error .Compared with other control techniques, the proposed Animated Oat Optimization based Fractional order PID controller achieves superior performance, with faster transient response, reduced steady-state error, and minimal total harmonic distortion, thereby improving capacitor voltage balancing and power quality in Packed U Cell Multilevel Inverter-based renewable energy systems. Also, the proposed Animated Oat Optimization based Fractional order PID controller is superior in terms of the convergence curve, as its value of the objective function reduces by 0.027 than the fitness value of Particle swarm optimization based FOPID , which is 0.047.

Despite the fact that deep reinforcement learning (RL) has surpassed human-level performances in various tasks, it still has several fundamental challenges. First, most RL methods require intensive data from the exploration of the environment to achieve satisfactory performance. Second, the use of neural networks in RL renders it hard to interpret the internals of the system in a way that humans can understand. To address these two challenges, we propose a framework that enables an RL agent to reason over its exploration process and distill high-level knowledge for effectively guiding its future explorations. Specifically, we propose a novel RL algorithm that learns high-level knowledge in the form of a finite reward automaton by using the L* learning algorithm. We prove that in episodic RL, a finite reward automaton can express any non-Markovian bounded reward functions with finitely many reward values and approximate any non-Markovian bounded reward function (with infinitely many reward values) with arbitrary precision. We also provide a lower bound for the episode length such that the proposed RL approach almost surely converges to an optimal policy in the limit. We test this approach on two RL environments with non-Markovian reward functions, choosing a variety of tasks with increasing complexity for each environment. We compare our algorithm with the state-of-the-art RL algorithms for non-Markovian reward functions, such as Joint Inference of Reward machines and Policies for RL (JIRP), Learning Reward Machine (LRM), and Proximal Policy Optimization (PPO2). Our results show that our algorithm converges to an optimal policy faster than other baseline methods.

One of the most common control algorithms in process control is the Proportional-Integral-Derivative (PID) controller, which could be very efficient if well tuned. The aim of this paper is to discuss the optimal tuning of the PID controller using extremum seeking (ES). Two ES algorithms, Perturbation Based Extremum Seeking (PBES) and Sliding Mode Based Extremum Seeking (SMBES), will be used to tune the PID parameters by searching for the optimal parameters that minimize a certain performance index. The Continuous Stirred Tank Reactor (CSTR), with irreversible first-order reaction, has been considered as a case study. The two algorithms have been implemented with two choices of Performance-Index (PI), Integral Absolute Error (IAE) criteria, and a combination of IAE with the integral of absolute of control signal. With the later choice of PI, both the algorithms show an improvement in the performance over the Internal Mode Control (IMC) tuning method, with good control behavior. The cost in the PBES tuning algorithm has converged to an ultimate bound around the optimal value within a lower number of iteration than the SMBES tuning algorithm. However, the speed of the SMBES algorithm can be increased but with the drawback of increasing the chattering effect produced in the PID parameters.

Predictive controllers are used in many in many industries recently. In this paper, a comparison between conventional Proportional plus integral (PI) controller using Ziegler–Nichols tuning method (Z–N) and predictive controllers such as dynamic matrix controller (DMC) and receding horizon control (RHC) are designed. Above-mentioned controllers are applied to a bench mark process quadruple-tank process (QTP) which is operated in nonminimum phase. The simulation for the process and above-mentioned controllers are obtained from MATLAB. Proposed controllers’ performance is supported with better tracking, disturbance rejection, and performance indices in terms of integral time absolute error (ITAE), integral absolute error (IAE) integral square error (ISE).KeywordsZiegler–Nichols tuning methodQuadruple-tank processDynamic matrix controllerReceding horizon controlRelative gain arrayNonminimum phase

Reinforcement learning and human behavior

Deep reinforcement learning (RL), where the agent learns from mistakes, has been successfully applied to a variety of tasks. With the aim of learning collision-free policies for unmanned vehicles, deep RL has been used for training with various types of data, such as colored images, depth images, and LiDAR point clouds, without the use of classic map--localize--plan approaches. However, existing methods are limited by their reliance on cameras and LiDAR devices, which have degraded sensing under adverse environmental conditions (e.g., smoky environments). In response, we propose the use of single-chip millimeter-wave (mmWave) radar, which is lightweight and inexpensive, for learning-based autonomous navigation. However, because mmWave radar signals are often noisy and sparse, we propose a cross-modal contrastive learning for representation (CM-CLR) method that maximizes the agreement between mmWave radar data and LiDAR data in the training stage. We evaluated our method in real-world robot compared with 1) a method with two separate networks using cross-modal generative reconstruction and an RL policy and 2) a baseline RL policy without cross-modal representation. Our proposed end-to-end deep RL policy with contrastive learning successfully navigated the robot through smoke-filled maze environments and achieved better performance compared with generative reconstruction methods, in which noisy artifact walls or obstacles were produced. All pretrained models and hardware settings are open access for reproducing this study and can be obtained at https://arg-nctu.github.io/projects/deeprl-mmWave.html

With the qualitative development of DC microgrids, the usage of different loads with unique conditions and features is now possible in electric power grids. Due to the negative impedance features of some loads, which are called constant power loads (CPLs), the control of DC power converters faces huge challenges from a stability point of view. Despite the significant advances in semiconductors, there is no upgrade in the control of gate drivers to exploit all potential of power electronic systems. In this paper, quantum computations are incorporated into artificial intelligence (AI) to stabilize a full-bridge (FB) DC-DC boost converter feeding CPL. Aiming to improve the bus voltage stabilization of the FB DC-DC boost converter, a quantum deep reinforcement learning (QDRL) control methodology is developed. By defining a reward function according to the specification of the FB power converter, the desired performance and control objectives are fulfilled. The main task of QDRL is to adjust the control coefficients embedded in the feedback controller to suppress the negative impedance effect resulting from deploying the CPLs. By deploying the potential advantages of quantum fundamentals, the deep reinforcement learning improved by quantum specifications will not only enhance the performance of the DRL algorithm on conventional processes but also advance related research areas such as quantum computing and AI. Unlike the basic quantum theory, which requires real quantum hardware, QDRL can be executed on classic computers. To examine the feasibility of the QDRL scheme, hardware-in-the-loop (HiL) examinations are conducted using the OPAL-RT. The comparison of the proposed controller with the classic state-of-the-art methodologies reveals the superiority and feasibility of QDRL-based control schemes in both the transient and steady-state conditions to other schemes. Analysis using various performance criteria, including the integral absolute error (IAE), integral time absolute error (ITAE), mean absolute error (MAE), and root mean square error (RMSE), demonstrates the dynamic improvement of the proposed scheme over sliding mode control (approximately 50%) and proportional integral control (approximately 100%).

It is difficult to give a robot all possible motions beforehand in a certain environment. Therefore, the robot needs to learn how to recognize other motions and to generate its own motions autonomously for working well. These learning algorithms need an efficient way to make recognition and generation of motions work together, because they take many computing resources. This paper focuses on a generation-based recognition. Our system consists of recognition and generation modules. The fanner and latter are constructed from left-to-right hidden Markov models (HMM) and reinforcement learning (RL), respectively. When a HMM in recognition module does not work enough, the model parameters of HMM are re-estimated by using a state-value function of RL in generation module. The proposed method enables us to improve the reliability of the HMM.

Gain-scheduled Composite Non linear Feedback (CNF) control has been implemented to Continuous Stirred Tank Reactor (CSTR) in this work. Chemical reaction takes place inside the CSTR, which is adjured from high residual concentration to low residual concentration. Control of CSTR is a bit complex, as it requires complete dynamic study and its implementation. During the entire process, the reaction kinetics changes from stable-unstable-stable range with the decrease in residual concentration. To stabilize the reaction, appropriate cooling is necessary, to ensure that overheating of the CSTR doesn't occur. Gain-scheduled composite non linear feedback controller has been implemented successfully for better tracking performance and error has been computed in terms of Integral of Absolute Error (IAE). The novelty of the work lies in lesser error obtained for this particular model of CSTR. Comparison of the IAE has been done with Model Reference Adaptive Control (MRAC) MIT control in CSTR.

Deep reinforcement learning (RL) methods often require many trials before convergence, and no direct interpretability of trained policies is provided. In order to achieve fast convergence and interpretability for the policy in RL, we propose a novel RL method for text-based games with a recent neuro-symbolic framework called Logical Neural Network, which can learn symbolic and interpretable rules in their differentiable network. The method is first to extract first-order logical facts from text observation and external word meaning network (ConceptNet), then train a policy in the network with directly interpretable logical operators. Our experimental results show RL training with the proposed method converges significantly faster than other state-of-the-art neuro-symbolic methods in a TextWorld benchmark.

This paper introduces a visual tracking control scheme for the two axes Inertially Stabilized Platform. An adaptive PID controller based on the Deep Deterministic Policy Gradient (DDPG) algorithm is used where it is treated as a Reinforcement Learning Policy. The optimal PID parameters are obtained at each time step via a training process. To effectively train the parameters, we address the problem of handling time delay and uncertainty of the system by presenting an observer system. Simulation results show that our method achieves good performance for various cases of time delay.

This paper presents a hybrid approach that integrates trajectory optimization (TO) and reinforcement learning (RL) for motion planning and control of free-flying multi-arm robots in on-orbit servicing scenarios. The proposed system integrates TO for generating feasible, efficient paths while accounting for dynamic and kinematic constraints, and RL for adaptive trajectory tracking under uncertainties. The multi-arm robot design, equipped with thrusters for precise body control, enables redundancy and stability in complex space operations. TO optimizes arm motions and thruster forces, reducing reliance on the arms for stabilization and enhancing maneuverability. RL further refines this by leveraging model-free control to adapt to dynamic interactions and disturbances. The experimental results validated through comprehensive simulations demonstrate the effectiveness and robustness of the proposed hybrid approach. Two case studies are explored: surface motion with initial contact and a free-floating scenario requiring surface approximation. In both cases, the hybrid method outperforms traditional strategies. In particular, the thrusters notably enhance motion smoothness, safety, and operational efficiency. The RL policy effectively tracks TO-generated trajectories, handling high-dimensional action spaces and dynamic mismatches. This integration of TO and RL combines the strengths of precise, task-specific planning with robust adaptability, ensuring high performance in the uncertain and dynamic conditions characteristic of space environments. By addressing challenges such as motion coupling, environmental disturbances, and dynamic control requirements, this framework establishes a strong foundation for advancing the autonomy and effectiveness of space robotic systems.

Reinforcement learning (RL) has attracted great interest from researchers in recent years. RL performs as human or better in many fields such as games and robot control. Although this technology is booming in computer science, it has not been practically applied in industrial process control. Up to now, proportional–integral–derivative (PID) control is still the most dominating and popular control method in industrial control. In this paper, we propose a combination of deep reinforcement learning (DRL) and PID control for better process control performance. The idea is generated by the following observations: for PID controller, its transient performance is not usually well enough to meet a strict requirement or in complex signal tracking tasks; For RL technology, a perfectly designed reward function is required for training. However, in practice, the reward function needs to be tested through trial and error, which will lead to a waste of computational power and time. By combining these two strategies, PID controller can help to improve the steady-state performance of RL control by its integral term, while the trained RL agent is able to improve the transient performance of PID controller. Several case studies with the water tank system are presented to demonstrate the effectiveness of the combined PID + RL control strategy.

An improved fractional-order sliding mode control (FOSMC) for PMSM is presented in this study to set the unavoidable parameters and to improve permanent magnet synchronous motors (PMSMs) drive performance, such as current and speed tracking accuracy. To determine the optimal parameters of the FOSMC for control speed in a PMSM drive, a neural network algorithm with reinforcement learning (RLNNA) is proposed. The FOSMC parameters are set by the ANN algorithm and then adapted through reinforcement learning to enhance the results. The proposed controller using RLNNA based on fractional-order sliding mode control (RLNNA-FOSMC) can drive the motor speed to achieve the referred value in a finite period of time, leading to faster convergence and improved tracking accuracy. For a fair comparison and evaluation, the proposed RLNNA-FOSMC is compared with conventional FOSMC by applying the integral of time multiplied absolute error as an objective function. The most commonly used objective functions in the literature were also compared, including the integral time multiplied square error, integral square error, and integral absolute error. To validate the performance of the RLNNA-FOSMC speed controller, different scenarios with different speeds steps were carried out. The computational results are promising and demonstrate the effectiveness of the proposed controller. Overall, the proposed RLNNA-FOSMC controller for the PMSM speed control system performed better than conventional FOSMC in numerical simulations.

Reinforcement learning (RL) can assist in medical decision making using patient data collected in electronic health record (EHR) systems. RL, a type of machine learning, can use these data to develop treatment policies. However, RL models are typically trained using imperfect retrospective EHR data. Therefore, if care is not taken in training, RL policies can propagate existing bias in healthcare. Literature that considers and addresses the issues of bias and fairness in sequential decision making are reviewed. The major themes to mitigate bias that emerge relate to (1) data management; (2) algorithmic design; and (3) clinical understanding of the resulting policies.