Reinforcement Learning-Based Control Strategies for Autonomous Robotic Systems

Fiber optic sensing (FOS) has become a well-known technology in response to the rising demands of the railway transportation field despite the abundance of electronic sensing systems in the market. FOS application boasts an all-in-one solution that is both efficient and versatile. In order to enhance the understanding of the capabilities of FOS, this paper presents a hybrid fiber optic sensing system with an improved sensing ability to facilitate transportation applications for primary or secondary security interfaces. The hybrid sensing scheme incorporates two different sensing systems designed for long-distance applications. The first system employs a coding technique for the transmitted pulses, which provide information on train location through cross-correlation with the reflected pulses from fiber Bragg grating (FBG) sensors located along the railway. The proposed system can accurately predict the train’s location up to a precision of one cm. The second system examines the wavelength drift of the reflected signal from the FBG sensor affected by the train using a tunable optical filter and photodetector. It determines essential parameters such as the train’s location, speed, and direction by measuring the Bragg wavelength shift and its direction. The effect of the train movement and speed on the applied strain on the FBG sensor is calculated in this work and applied to the simulation to determine the train’s location, speed, and direction. A calibration table facilitates the correlation between the train speed and the shift in the FBG center wavelength, which helps ensure accurate results. The hybrid fiber optic sensing system is designed to facilitate railway transportation applications’ sustainability and security.

Introduction: Deep reinforcement learning (deep RL) integrates the principles of reinforcement learning with deep neural networks, enabling agents to excel in diverse tasks ranging from playing board games such as Go and Chess to controlling robotic systems and autonomous vehicles. By leveraging foundational concepts of value functions, policy optimization, and temporal difference methods, deep RL has rapidly evolved and found applications in areas such as gaming, robotics, finance, and healthcare. Objective: This paper seeks to provide a comprehensive yet accessible overview of the evolution of deep RL and its leading algorithms. It aims to serve both as an introduction for newcomers to the field and as a practical guide for those seeking to select the most appropriate methods for specific problem domains. Methods: We begin by outlining fundamental reinforcement learning principles, followed by an exploration of early tabular Q-learning methods. We then trace the historical development of deep RL, highlighting key milestones such as the advent of deep Q-networks (DQN). The survey extends to policy gradient methods, actor–critic architectures, and state-of-the-art algorithms such as proximal policy optimization, soft actor–critic, and emerging model-based approaches. Throughout, we discuss the current challenges facing deep RL, including issues of sample efficiency, interpretability, and safety, as well as open research questions involving large-scale training, hierarchical architectures, and multi-task learning. Results: Our analysis demonstrates how critical breakthroughs have driven deep RL into increasingly complex application domains. We highlight existing limitations and ongoing bottlenecks, such as high data requirements and the need for more transparent, ethically aligned systems. Finally, we survey potential future directions, highlighting the importance of reliability and ethical considerations for real-world deployments.

Reinforcement learning-based real-time optimization of friction stir welding parameters for copper–aluminium dissimilar interfaces

While the recent advanced military operational concept requires an intelligent support of command and control, Reinforcement Learning (RL) has not been actively studied in the military domain. This study points out the limitations of RL for military applications from literature review and aims at improving the understanding of RL for military decision support under the limitations. Most of all, the black box characteristic of Deep RL makes the internal process difficult to understand in addition to complex simulation tools. A scalable weapon selection RL framework is built which can be solved either by a tabular form or a neural network form. The transition of the Deep Q-Network (DQN) solution to the tabular form makes it easier to compare the result to the Q-learning solution. Furthermore, rather than using one or two RL models selectively as before, RL models are divided as an actor and a critic, and systematically compared. A random agent, Q-learning and DQN agents as a critic, a Policy Gradient (PG) agent as an actor, Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) agents as an actor-critic approach are designed, trained, and tested. The performance results show that the trained DQN and PPO agents are the best decision supporter candidates for the weapon selection RL framework.

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

In this paper, one large Fiber Bragg Grating (FBG) sensor sensing network for bridge local health monitoring was studied, designed and implemented to Harbin Sifangtai Bridge, China. According to the hot point principle and finite element analysis results, FBG sensors installation scheme was made to monitor the local strain of key elements, respectively. 60 FBG sensors had been installed in monitoring points along the height of I-shape girders distributed in three main monitoring sections along the bridge. Monitoring results in load testing period was obtained, the static and dynamic results show that FBG sensors can monitoring the local strain precisely, and the local strain response is in the safe range under test vehicle load. Monitoring results in service period was obtained, 9 months after load test. Results show that maximum local strain responses in three main monitoring sections are in the safe range, under the dynamic vehicle load. Comparison of the monitoring results from FBG sensor sensing network are used to unveil the local health condition of Sifangtai Bridge reflected by local strain response.

Embedded fiber Bragg grating (FBG) sensors are attractive for in-situ structural monitoring, especially in fiber reinforced composites. Their implementation in metallic structures is hindered by the thermal limit of the protective coating, typically a polymer material. The purpose of this study is to demonstrate the embedding of FBG sensors into metals with the ultimate objective of using FBG sensors for structural health monitoring of metallic structures. To that end, ultrasonic additive manufacturing (UAM) is utilized. UAM is a solid-state manufacturing process based on ultrasonic metal welding that allows for layered addition of metallic foils without melting. Embedding FBGs through UAM is shown to result in total cross-sectional encapsulation of the sensors within the metal matrix, which encourages uniform strain transfer. Since the UAM process takes place at essentially room temperature, the industry standard acrylate protective coating can be used rather than requiring a new coating applied to the FBGs prior to embedment. Measurements presented in this paper show that UAM-embedded FBG sensors accurately track strain at temperatures higher than 400 °C. The data reveals the conditions under which detrimental wavelength hopping takes place due to non-uniformity of the load transferred to the FBG. Further, optical cross-sectioning of the test specimens shows inhibition of the thermal degradation of the protective coating. It is hypothesized that the lack of an atmosphere around the fully-encapsulated FBGs makes it possible to operate the sensors at temperatures well above what has been possible until now. Embedded FBGs were shown to retain their coatings when subjected to a thermal loading that would result in over 50 percent degradation (by volume and mass) in atmospherically exposed fiber.

This study aims to systematically compare the performance of two deep reinforcement learning algorithms – Proximal Policy Optimization (PPO) and Deep Q-Network (DQN) – across different game environments. To achieve this, eight distinct test environments from the OpenAI Gymnasium library (CartPole-v1, FrozenLake-v1, LunarLander-v3, Taxi-v3, MountainCar-v0, Blackjack-v1, CliffWalking-v0, and Acrobot-v1) were utilized. Each environment was trained over 1,000,000 timesteps. For each algorithm, key performance metrics such as average reward, training time, standard deviation, success rate, and the highest and lowest reward values were calculated and visualized through graphs. Additionally, the strengths and weaknesses of the algorithms in different environments were analyzed. The results indicate that PPO performs more consistently and effectively in tasks requiring continuous actions, whereas DQN achieves faster and more reliable outcomes in deterministic environments with discrete action spaces. This study provides meaningful insights by comparing the performance of PPO and DQN under identical conditions, while most prior research has examined these algorithms separately.

The article describes process of implementation of optical Fiber Bragg Grating (FBG) sensors into the composite wing structure and their behavior during the strength test. The wing is of all-composite construction. The upper and lower skins are made of glass/epoxy composite. The spar caps are made of carbon/epoxy unidirectional composite. Optical fibers were integrated directly into the spar caps and into the adhesive joints. They were oriented in parallel with the main spar axis. The first optical fiber with chain of multiple FBG sensors was integrated into the structure of upper spar cap. Another FBG chain of FBG sensors was located in the adhesive joint of lower spar cap and shear web. The wing was instrumented with strain gages as well. Strain gages were glued to the sides of the spar caps. Static load was produced by a hydraulic actuator. Experimental results from strain gages and FBG sensors were compared with the results of the analytical analysis of the wing.

Fiber Bragg Grating (FBG) sensor is a new type of optical sensor that is light in quality, small in volume, easy to achieve distributed measurement, and easy to embed. When FBG sensor technology is introduced to the damage detection of the CFRP shaft, the shortcomings of conventional damage detection technologies, including ultrasonic testing and radiographic testing, can be effectively overcome. Considering the working load, operating environment and life requirements of the CFRP shaft , combined with the relevant national and industry standards for composite material testing, Prepare three CFRP shaft with embedded fiber gratings (non-destructive CFRP shaft, single hole CFRP shaft , double hole CFRP shaft), the hole diameter is 1mm, and the layering of CFRP shaft is ([0 / 90 / 45 / 0 / 0 / 45 / -45 / 0 / -45 / 0 / 90 / 90 / 0 / -45 / 0 / -45 / 45 / 0 / 0 / 45 / 90 / 0 ]). FBG is embedded between 18 and 19 layers, pasted on the surface of CFRP shaft with 10 FBG sensors (named FBG1, FBG2... FBG10). The strain information of the CFRP shaft is obtained through fiber grating to obtain the deformation of the CFRP shaft under a three-point bending load. Each CFRP shaft is loaded 12 times (A1, A2, A3, B1, B2, B3, C1, C2, C3, D1, D2, D3). Through the BP neural network method, the mapping relationship between strain distribution and damage model was established. In the model, the FBG sensor measures the strain of the CFRP shaft and forms the input of the neural network. Combined with the damage situation of the composite material, the learning samples of the network formed. After learning and training, the neural network can damage recognition. To better verify the accuracy of the neural network to distinguish and identify the CFRP shaft non-destructive, single-hole, and double-hole damage patterns. The finite element analysis was used to simulate the damage of three CFRP shafts under the same experimental conditions, and extract the eigenvalues at the same position to form the input of the network, combined with the damage situation of the composite material, constitutes a learning sample of the network. The experiment shows that the CFRP shaft experiment based on BP neural network verifies the reliability of the CFRP shaft damage recognition method. The finite element analysis is used to extract the strain information of CFRP shaft measuring points with different damage modes. The damage position information and strain information were input into the BP neural network. The experimental results are in good agreement with the simulation results, which verifies the accuracy of the BP neural network to distinguish the CFRP shaft non-destructive, single-hole, and doublehole damage patterns.

Fiber Bragg grating (FBG) sensors are excellent transducers for collecting ultrasonic wave signals for structural health monitoring (SHM). Typically, FBG sensors are directly bonded to the surface of a structure to detect signals. Unfortunately, demodulating relevant information from the collected signal demands a high signal-to-noise ratio because the structural ultrasonic waves have low amplitudes. Our previous experimental work demonstrated that the optical fiber could be bonded at a distance away from the FBG location, referred to here as remote bonding. This remote bonding technique increased the output signal amplitude compared to the direct bonding case, however the mechanism causing the increase was not explored. In this work, we simulate the previous experimental work through transient analysis based on the finite element method, and the output FBG response is calculated through the transfer matrix method. The model is first constructed without an adhesive to assume an ideal bonding condition, investigating the difference in excitation signal coherence along the FBG length between the two bonding configurations. A second model is constructed with an adhesive to investigate the effect of the presence of the adhesive around the FBG. The results demonstrate that the amplitude increase is originated not from the excitation signal coherence, but from the shear lag effect which causes immature signal amplitude development in the direct bonding case compared to the remote bonding case. The results also indicate that depending on the adhesive properties the surface-bonded optical fiber manifests varying resonant frequency, therefore resulting in a peak amplitude response when the input excitation frequency is matched. This work provides beneficial reference for selecting adhesive and calibrating sensing system for maximum ultrasonic detection sensitivity using the FBG sensor.

This paper proposes multipoint optical fiber sensing system based on fiber Bragg grating (FBG) sensor array. The low reflectivity FBG sensors having the same Bragg wavelength are connected cascaded in the same fiber line as the sensing elements of optical fiber sensing system. This can increase the monitoring points and reduce manufacturing costs of FBG sensor. To verify the effectiveness, the system simulator based on equivalent circuit model by microwave technology was developed. Simulation result shows that though multi-reflections can be caused by FBG sensors with the same Bragg wavelength connected cascaded in the same line, the effect may become weak as reflectivity of FBG sensors are reduced to 0.02.

Thermal modeling of the power transformer to estimate the internal temperature only reflects the winding hot spot to a certain extent. The application of optical fiber sensors to measure transformer internal temperature directly can obtain the true temperature with high accuracy and stability and overcome the shortcomings of traditional electric sensors. This paper analyses the measuring principle of FBG (Fiber Bragg Grating) sensor, and constructs transformer internal temperature measurement platform with FBG temperature sensors, then adopt short-circuit method to simulate the process of winding heat under different load conditions. Finally the measurement of transformer internal temperature is conducted respectively with FBG temperature sensors and thermocouples. Comparison and analysis of the results shows that FBG sensors can effectively measure the internal temperature of the transformer.

Railway tracks are the direct supporting structures of the trains, which are vulnerable to produce large deformation under the temperature stress or subgrade settlement. The health status of track is critical, and the track should be routinely monitored to improve safety, lower the risk of excess deformation and provide reliable maintenance strategy. In this paper, the distributed optical fiber sensor was proposed to monitor the continuous deformation of the track. In order to validate the feasibility of the monitoring method, two deformation monitoring tests on one steel rail model in laboratory and on one real railway tack in outdoor were conducted respectively. In the model test, the working conditions of simply supported beam and continuous beam in the rail model under several concentrated loads were set to simulate different stress conditions of the real rail, respectively. In order to evaluate the monitoring accuracy, one distributed optical fiber sensor and one fiber Bragg grating (FBG) sensor were installed on the lower surface of the rail model, the strain measured by FBG sensor and the strain calculated from FEA were taken as measurement references. The model test results show that the strain measured by distributed optical fiber sensor has a good agreement with those measured by FBG sensor and FEA. In the outdoor test, the real track suffered from displacement and temperature loads. The distributed optical fiber sensor installed on the rail can monitor the corresponding strain and temperature with a good accuracy.

In this study, measurements from low-impact velocity experiments including embedded and surface mounted optical fiber Bragg grating (FBG) sensors were used to obtain detailed information pertaining to damage progression in two-dimensional laminate woven composites. The woven composites were subjected to multiple strikes at 2m/s until perforation occurred, and the impactor position and acceleration were monitored throughout each event. From these measurements, we obtained dissipated energies and contact forces. The FBG sensors were embedded and surface mounted at different critical locations near penetration-induced damaged regions. These FBG sensors were used to obtain initial residual strains and axial and transverse strains that correspond to matrix cracking and delamination. The transmission and reflection spectra were continuously monitored throughout the loading cycles. They were used, in combination with the peak contact forces, to delineate repeatable sensor responses corresponding to material failure. From the FBG spectra, fiber and matrix damage were separated by an analysis based on signal intensity, the presence of cladding modes, and the behavior of individual Bragg peaks as a function of evolving and repeated impact loads. This provided an independent feedback on the integrity of the Bragg gratings. A comparison by number of impact strikes and dissipated energies corresponding to material perforation indicates that embedding these sensors did not affect the integrity of the woven systems and that these measurements can provide accurate failure strains.