Multi-fidelity reinforcement learning with control variates

Multi-fidelity surrogate (MFS) modeling technology, which efficiently constructs surrogate models using low-fidelity (LF) and high-fidelity (HF) data, has been studied to enhance the predictive capability of engineering performances. In addition, several neural network (NN) structures for MFS modeling have been introduced, benefiting from recent developments in deep learning research. However, existing multi-fidelity (MF) NNs have been developed assuming identical sets of input variables for LF and HF data, a condition that is often not met in practical engineering systems. Therefore, this study proposes a new structure of composite NN designed for MF data with different input variables. The proposed network structure includes an input mapping network that connects the LF and HF data's input variables. Even when the physical relationship between these variables is unknown, the input mapping network can be concurrently trained during the process of training the whole network model. Customized loss functions and activation variables are suggested in this study to facilitate forward and backward propagation for the proposed NN structures when training MF data with different inputs. The effectiveness of the proposed method, in terms of prediction accuracy, is demonstrated through mathematical examples and practical engineering problems related to tire performances. The results confirm that the proposed method offers better accuracy than existing surrogate models in most problems. Moreover, the proposed method proves advantageous for surrogate modeling of nonlinear or discrete functions, a characteristic feature of NN-based methods.

Design approach for tilt propellers of UAM/eVTOLs for cruise and hover considering aerodynamic and aeroacoustic characteristics via a multi-fidelity model

The growing demand for biopharmaceutical products reflects their effectiveness in medical treatments. However, developing new biopharmaceuticals remains a major bottleneck, often taking up to a decade before market approval. Machine learning (ML) models have the potential to accelerate this process, but their success depends on access to large and diverse data sets for training. Multi-fidelity ML techniques offer a promising solution by integrating abundant, low-cost, and less accurate low-fidelity (LF) data with limited, expensive, and more accurate high-fidelity (HF) data. In this framework, LF data capture global system trends, while HF data refine and align model predictions with the available ground truth. Such integration can substantially reduce development costs and timelines by minimizing the need to acquire HF data, for example, through extensive experimental campaigns. This work reviews developments in surrogate modeling within the biopharmaceutical context, including Gaussian processes, neural networks, and physics-informed approaches. It also provides practical recommendations for identifying appropriate LF and HF data. Existing research has primarily focused on upstream processing and drug discovery, highlighting opportunities to extend these methods to other stages, like downstream processing. While Gaussian processes and neural networks remain the most frequently used models, emerging architectures such as Transformer and diffusion models present promising directions for future research.

In subsurface flow simulation, data-driven deep learning surrogate models have emerged as a promising alternative to traditional simulation methods. However, a major challenge is the large amount of high-fidelity simulation data required for training, and purely data-driven flow surrogate models often lack theoretical support, limiting their applicability to engineering problems. In this study, we propose an efficient surrogate model construction method that integrates physical constraints and multi-fidelity training data. The process begins with pre-training a deep learning model using a large amount of low-cost, low-fidelity data to effectively initialize network parameters. The pre-trained model is then fine-tuned with a smaller amount of high-fidelity data. Throughout the training, physical losses constrain the model to adhere to physical principles. For the case considered here, this approach reduces the need for high-fidelity data by approximately 83% compared to traditional training methods without compromising predictive accuracy. Additionally, incorporating physical loss during model training e ffectively enhances the surrogate model's predictive accuracy, with physical loss reduced by nearly 70% compared to purely data-driven training. The method employs the Jacobian matrix to aid in model training, significantly reducing the computational cost of physical losses. Overall, this surrogate model construction method addresses the challenges of obtaining training data and the lack of physical support in surrogate models, providing reliable support for complex reservoir development decisions.

Summary In the field of subsurface flow simulation, deep-learning-based surrogate modeling is shown as a promising approach to significantly reduce the computational cost associated with full-physics reservoir simulations. However, the successful construction of highly accurate deep-learning-based models often requires a large number of training simulations, which can be very time-consuming to generate by itself for large-scale systems. The incorporation of physics constraints during the training process was shown to be an effective approach to reduce the training cost and to improve model accuracy. However, it remains challenging to reimplement the physics constraints correctly for modeling training when it comes to reservoir models with complex physics or gridding methods. In this study, we propose a physics-constrained surrogate (PCS) model training approach for production optimization that integrates multifidelity training data, together with a new way of implementing the physics constraints that leverages an existing mature reservoir simulator. The proposed approach starts with model pretraining on a relatively large number of low-cost, low-fidelity training data, followed by model fine-tuning with a smaller number of high-fidelity training samples and the inclusion of physics constraints. The physics constraints are implemented by adding the residuals of discretized governing equations into the loss function. Training with multifidelity data and incorporating physics constraints allows for reduced reliance on high-fidelity data while enhancing physical consistency. Systematic comparison studies were performed for both 2D and 3D cases, and it was shown that the proposed approach can reduce the computational costs associated with training data generation by about 80%, while achieving a similar level of prediction accuracy of the surrogate model. In addition, under a small number of training simulations (i.e., 50 equivalent high-fidelity runs), our proposed PCS model with multifidelity data can reduce the prediction error by 90%, in comparison with the model trained with only high-fidelity data. Finally, the trained surrogate model was applied to a well-control optimization problem. In comparison with the use of full-order simulations, the total computational time can be reduced by 97.7%.

A physics-informed diffusion model for high-fidelity flow field reconstruction

Many researchers have studied Multi-fidelity (MF) models to obtain the optimum solution efficiently when the time of analysis and evaluation is long. The MF model is a meta-model that combines High fidelity (HF) data, which requires large computational cost for evaluation; and Low fidelity (LF) data, which needs small computation cost but has low accuracy. Therefore, integrating HF and LF data with a scale factor and correction model is highly important in MF modeling. This study investigates the performance of the MF model in terms of definition and estimation. First, three different MF models are built: An LF model multiplied by the scale factor, an LF model combined with a correction model, and an LF model with both scale factor and correction model. Second, the effects of scale factor on the MF model are analyzed in terms of constant vs. linear and maximum likelihood estimation vs. least squares estimation. Finally, a correction model is built using two methods, namely, interpolation and regression, to assess the influence of the correction model on the MF model. To evaluate the MF model performance, several test problems are applied, and the root mean square error of each model is estimated as the accuracy measure. In conclusion, the characteristics of different types of MF models are summarized and guidelines for the generation of MF models are proposed to approximate closely the precise models.

The n-octanol/buffer solution distribution coefficient at pH = 7.4 (log D7.4) is an indicator of lipophilicity, and it influences a wide variety of absorption, distribution, metabolism, excretion, and toxicity (ADMET) properties and druggability of compounds. In log D7.4 prediction, graph neural networks (GNNs) can uncover subtle structure-property relationships (SPRs) by automatically extracting features from molecular graphs that facilitate the learning of SPRs, but their performances are often limited by the small size of available datasets. Herein, we present a transfer learning strategy called pretraining on computational data and then fine-tuning on experimental data (PCFE) to fully exploit the predictive potential of GNNs. PCFE works by pretraining a GNN model on 1.71 million computational log D data (low-fidelity data) and then fine-tuning it on 19,155 experimental log D7.4 data (high-fidelity data). The experiments for three GNN architectures (graph convolutional network (GCN), graph attention network (GAT), and Attentive FP) demonstrated the effectiveness of PCFE in improving GNNs for log D7.4 predictions. Moreover, the optimal PCFE-trained GNN model (cx-Attentive FP, Rtest2 = 0.909) outperformed four excellent descriptor-based models (random forest (RF), gradient boosting (GB), support vector machine (SVM), and extreme gradient boosting (XGBoost)). The robustness of the cx-Attentive FP model was also confirmed by evaluating the models with different training data sizes and dataset splitting strategies. Therefore, we developed a webserver and defined the applicability domain for this model. The webserver (http://tools.scbdd.com/chemlogd/) provides free log D7.4 prediction services. In addition, the important descriptors for log D7.4 were detected by the Shapley additive explanations (SHAP) method, and the most relevant substructures of log D7.4 were identified by the attention mechanism. Finally, the matched molecular pair analysis (MMPA) was performed to summarize the contributions of common chemical substituents to log D7.4, including a variety of hydrocarbon groups, halogen groups, heteroatoms, and polar groups. In conclusion, we believe that the cx-Attentive FP model can serve as a reliable tool to predict log D7.4 and hope that pretraining on low-fidelity data can help GNNs make accurate predictions of other endpoints in drug discovery.

The modeling of wake effects plays an essential role in wind farm optimal design and operation. In this study, a novel deep learning method, called Super-Fidelity Network (SFNet), is proposed for wind farm wake modeling, which would be the first attempt to combine the advantages of both analytical models and numerical models through deep learning methods. Specifically, the low-fidelity flow fields generated by the analytical models serve as the prior information for predicting high-fidelity flow fields. Then the SFNet learns the mapping relationships between low-fidelity data and high-fidelity data, thereby predicting high-fidelity flow fields without resorting to huge computational resources. Numerical experiments demonstrate that the mean absolute error of the developed model is just 1.9% with respect to the freestream wind speed when compared with high-fidelity data, after trained on only 45 samples. In addition, the generalizability of the proposed SFNet in yaw angles, wind speeds and array column extensions is verified by a series of numerical experiments. Furthermore, the experimental results demonstrate that the trained model is able to predict the flow field of a wind farm consisting of 100 turbines within several seconds based on a standard desktop.

Since the physical structure and mathematical models are more complex, reliability analysis in practical engineering can be expensive and difficult. A two-level multifidelity metamodel method for reliability analysis is introduced. Following the surrogate model in most of the relevant works, low-fidelity data and high-fidelity data are integrated by co-Kriging model. Besides, the co-Kriging model also provide an approximation for initial performance function. Bayesian method is adopted in model solution and a hybrid Markov chain Monte Carlo (MCMC) sampling algorithm is proposed. High-fidelity response of reliability performance function is estimated by the conditional distribution derivation based on Bayesian theory. Failure domain is identified by indicator function in sampling space that consists of samples derived from MCMC. Accordingly, failure probability estimations are obtained using Monte Carlo simulation (MCS). It is demonstrated through an illustrative example that the proposed method is valid and accurate.

Transfer learning based multi-fidelity physics informed deep neural network

A multi-fidelity transfer learning strategy based on multi-channel fusion

This thesis aims (i) to better understand near-field interactions in magnetic colloids, (ii) to study the particle distribution pattern effects on the magnetic response of magnetically labeled ceramic particles, and (iii) to apply these learnings to process ceramic-polymer composites with anisotropic micro-structures enabling a better understanding of fundamental structure-property relationships of fracture toughness in anisotropic composites. This work first focuses on predicting the magnetostatic interactions of super- paramagnetic particles in a transient simulation by applying machine learning to this field for the first time. Conventional predictive models for magnetostatic interactions include the widely applied and analytic dipole model, a matrix-based mutual dipole model and a numerically intensive but precise finite element model. Due to the accuracy and computational challenges of the near- field magnetic interaction with high susceptibility particles, and inspired by the success of machine learning approach in complex problems such as solid mechanics and hydrodynamics, we hypothesized that machine learning would deliver an accurate and fast predictive model that would enable large scale simulations. Simple neural network machine learning models showed limitations. To overcome this, we combined the fast computation advantage of dipole model (low fidelity data) with the enhanced accuracy of finite element modeling (high fidelity data). A multi-fidelity neural network model is built on this database and found to predict magnetostatic interactions in simple particle systems extremely well. However, this effort uncovered some limitations and concerns with the machine learning approach. The accuracy, benefits and limitations of this approach are discussed in this work. One of the motivations for developing a predictive machine learning code was to be able to model ceramic microparticles that are coated with tens of thousands of magnetic nanoparticles that have found many applications in composites, active optics, and micro- rheology. These labeled ceramics dramatically orient in applied magnetic fields with surprising speed. The discrete model developed in this thesis work is applied to model these massive multi-body systems and compared to previously proposed continuum analytic models that treat the labelled particles as magnetic ellipsoidal shells. In our discrete model, we employ the Monte Carlo method to find the average response of randomized assemblies of tens of thousands of magnetic nanoparticles adsorbed to the ceramic surface. The combined discrete model and Monte Carlo approach brings into the calculation the correct first principles of magnetostatic interactions between magnetic nanoparticles in the coating layer. This approach, therefore, provides a means for measuring the influence on the magnetic response of the platelet brought on the distribution pattern of the nanoparticles that can be experimentally tuned through controlled aggregation. These developed methodologies allow us to probe many of our hypotheses able these magnetic nanoparticle labelled ceramics including the root cause of their dramatic responses. The magnetic response of the labelled ceramic platelets can be applied to manipulate the micro-structure of ceramic-polymer composites to create fully aligned composites that exhibit toughness with sufficient enough control to establish new fundamental understandings in structure-property relationships in this field. The thesis turns attention to these systems and discusses processing, structure, and property paradigms within this class of materials. First, the process of manufacturing alumina reinforced acrylate polymer composites is discussed. Next, the experimental characterization of their mechanical properties is detailed. Finally, details about resultant fracture toughness properties are discussed including the crack kinking behavior in thin samples with anisotropic fracture energy. The effects of non-singular stress at the front of the crack tip associated with the stabilizing/destabilizing effect on the crack kinking will be discussed under the linear elastic fracture mechanics framework using plane stress assumptions. A phase-field simulation result carried out by collaborators will also be presented. Ultimately, we will show that the fracture energy anisotropy and the non-singular stress that can be modulated by the process zone size will work together to predict the existence of an anisotropic threshold for crack kinking and how this threshold is strongly dependent on load distribution.

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

The next generation low-cost modular unmanned combat aerial vehicles (UCAVs) provide the opportunity to implement innovative solutions to complex tasks, while also bringing new challenges in design, production, and certification subjects. Solving these problems with tools that provide fast modeling in line with the digital twin concept is possible. In this work, we develop an artificial intelligence (AI) based multifidelity surrogate model to determine performance parameters of innovative modular UCAVs. First, we develop a data generation algorithm that includes a high-fidelity model based on computational fluid dynamics methods and a low-fidelity model based on computational aerodynamic approaches. In the next step, a new transfer learning-based surrogate model is generated using multifidelity data. Thanks to this approach, the developed AI model more accurately predicted the flow conditions that were missing in the high-fidelity data with the data obtained from the low-order model. The performance of the proposed AI-based surrogate model is to be investigated in terms of accuracy, robustness, and computational cost using a generic modular UCAV configuration.