A nonlocal physics-informed deep learning framework using the peridynamic differential operator

In recent years, successful applications of deep learning (DL) have inspired scientists to explore the possibilities of applying DL approaches to modeling scientific problems. Existing studies have revealed that to bake the physics into the DL makes a good supplement to the traditional numerical methods (e.g., finite element, finite volume method) which primarily rely on partial differential equations (PDEs). While DL models are ordinarily trained in a purely data-driven manner, integrating physics into them for simulating scientific problems has several benefits such as (i) physics constraints could regularize the over-parameterized model and hence mitigate the overfitting issue commonly seen in DL; (ii) physics information could also effectively reduce the amount of data needed for training the model; (iii) the resultant physics-informed DL models feature better interpretability and generalizability compared with the conventional black-box model. Furthermore, the powerful expressiveness of the deep network, guaranteed by the universal approximation theorem, makes it a suitable approximator for the solution to a physical system. In this dissertation, we develop two different DL architectures (or approaches), one being continuous scheme-based while the other discrete scheme-based, that leverage physics knowledge for modeling scientific problems. Through comprehensive numerical experiments, we demonstrate the proposed models can be used in solving general PDEs, establishing predictive data-driven models for dynamical systems, identifying the parameters in governing PDEs or even discovering the entire governing PDEs of dynamical systems from scarce and noisy measurements. The continuous model roots on the physics-informed neural network (PINN) which uses a fully connected neural network (FCNN) to approximate the physical fields of a system globally. This model is mesh-free as the residual of the physics (e.g., PDEs, initial/boundary values) is evaluated on a set of collocation points within the physical domain. Several applications including the forward simulations, data-driven simulations and solving inverse problems are presented to exemplify the advantages of PINN over traditional numerical methods. However, the original PINN suffers from inaccurate initial/boundary values due to the weak enforcement of the initial/boundary conditions (I/BCs). To overcome this issue, we propose an improved PINN model by utilizing multiple deep neural networks (DNNs) to construct the solution. Through a DNN pre-trained to represent the initial/boundary values, the approximated solution would obey the given I/BCs forcibly. With several numerical examples, we show that the improved PINN is characterized with much better accuracy on the I/BCs. Though the PINN shows great promise in data-driven modeling and solving inverse problems, some inherent limitations of PINN still exist, such as (i) the solution might lacks fine-scale details due to the global approximation of FCNN; (ii) high computational expense caused by the FCNN it roots on; (iii) incapability to incorporate existing PDE terms (e.g., $\Delta u$) into the network architecture. To overcome these drawbacks, this dissertation also proposes a discrete model - Physics-encoded Recurrent Convolutional Neural Network (PeRCNN) which recurrently updates the solution (or state variable) for time marching. Specifically, it utilizes convolutional neural network (CNN) to capture the spatial patterns of the solution while the recurrent network mimics the forward Euler scheme (or Runge-Kutta scheme) in numerical methods. PeRCNN is a mesh-based and discrete model due to the discretization in time and spatial dimension. The local connectivity of CNN makes PeRCNN more computationally efficient. In addition, the coercive encoding mechanism of physics in PeRCNN, fundamentally different from the PINN relying on soft penalty, ensures the network to rigorously obey given physics. The proposed PeRCNN is successfully applied to solving general PDEs, the data-driven modeling of dynamical systems and the data-driven discovery of governing PDEs from scarce and noisy measurements. Comparisons with the state-of-the-art DL models demonstrate that the proposed PeRCNN possesses excellent computational efficiency, accuracy and generalizability. --Author's abstract

As Li-ion batteries become more essential in today's economy, tools need to be developed to accurately and rapidly diagnose a battery's internal state-of-health. Using a Li-ion battery's (high-rate) voltage response, it is proposed to determine a battery's internal state through Bayesian calibration.However, Bayesian calibration is notoriously slow and requires thousands of model runs. To accelerate parameter inference using Bayesian calibration, a surrogate model is developed to replace the underlying physics-based Li-ion model. Developing a surrogate model for rapid Bayesian calibration analysis is discussed for both the single particle model (SPM) and the pseudo two-dimensional (P2D) model.Surrogate models are constructed using physics-informed neural networks (PINNs) that encode the influence of internal properties on observed voltage responses. In practice, a neural network can be trained by: 1) using simulation results of the physics-based model (i.e., a data-loss approach); 2) using the residuals of the governing equations themselves (i.e., a physics-loss approach); or 3) using a combination of simulation results and governing equation residuals. In the present work, PINNs are developed using a variety of training losses and neural network architectures. In this analysis, it is shown that a PINN surrogate model can be reliably trained with only physics-informed loss. However, using a coupled data-informed and physics-loss approach produced the most accurate PINNs. Figure~\\ref{fig:spm_2d} illustrates the absolute relative errors of trained PINN networks using several different training losses and neural network architectures.After determining a consistent training strategy for both the SPM and P2D PINN surrogate models, the PINNs are extended to determine additional internal state-of-health parameters. As more and more parameters were introduced, the PINN training suffered from ``the curse of dimensionality", which was mitigated by using a hierarchical training approach (where a PINN trained with fewer variable model parameters was used to train a PINN with more variable model parameters).Next, the high-dimensionality PINN surrogates are then integrated into Bayesian calibration schemes to identify internal Li-ion battery properties from experimentally measured voltages. Interpreting the high-dimensional parameter posteriors is discussed with respect to model error, parameter prior choices, and experimental errors. Figure 1

3D concrete printing (3DCP) is an innovative construction method that extrudes cementitious materials layer-by-layer to fabricate building components based on a digital model. 3DCP has gained increasing adoption globally for projects like buildings, bridges, retaining walls, and stormwater management systems. However, 3DCP places stringent demands on the rheological properties of the printable cementitious materials. Rheology is critical for the flow and buildability of fresh concrete during extrusion. The mixture must have a low yield stress for pumping, moderate viscosity to hold its shape after deposition, minimal bleeding and segregation of aggregates, as well as responsive rheology that can transition from fluid to solid state as layers are printed sequentially. In this dissertation, we aim to solve the challenges mentioned above including: (1) accurate and efficient quantification of rheological and thixotropic properties of cementitious materials, (2) evaluation of flow behavior of fresh concrete during extrusion. Using traditional experimental methods to evaluate rheological properties of cement paste is labor-intensive and time-consuming. Numerical simulations like Finite Element Method (FEM) or Computational Fluid Dynamics (CFD) can help increase the speed of finding the most suitable material for 3DCP. However, these simulation methods are costly in computing power, making them less desirable for simulating the dynamic and complex process of 3DCP. Moreover, mesh-based approaches are not suitable for complex geometries as the mesh generation would be challenging with curved boundaries, small features, holes or thin regions. Physics-Informed Neural Network (PINN) is introduced in the work as an innovative solution to simulate and model the whole process of 3DCP. It can leverage the flexibility and computational efficiency of data-based neural networks, which can accelerate the learning and convergence speed. At the same time, Partial Differential Equations (PDEs) would be embedded in the network structure to increase the accuracy of its prediction. The integrated governing physical laws can help the neural network understanding the underlying physics of the 3DCP process, potentially increasing the accuracy and reliability of simulations. A significant part of the research is dedicated to understanding the rheological and thixotropic behavior of cementitious materials, which is critical for the material design in 3DCP. We developed a Rheology-informed Neural Network, RheologyNet, where we embedded the rheological constitutive laws into the loss function. These physical laws can help the network focus on simulating the rheological properties of cementitious materials. Moreover, a thixotropic evaluation system has been established based on the proposed model to evaluate the thixotropic property of cement paste. This study is presented in Chapter 2. Then, we proposed a Navier-Stokes Informed Neural Network (NSINN) to study the flow behavior of cement paste in the 3DCP barrel and nozzle since it involves complicated Multiscale or Multiphysics behaviors, such as anisotropy and non-uniform shear rate distribution. The Navier Stokes Equation and the rheological constitutive equations are coupled and embedded into the NSINN architecture to simulate the flow behavior in the barrel during 3D-printing extrusion process. This approach demonstrates the potential of PINNs to provide high-accuracy simulations of velocity and pressure fields in a computationally efficient manner compared to traditional mesh-based models. Also, we used the NSINN to learn the relationship between the nozzle size and the printing quality. The details of the study are presented in Chapter 3. At last, a multi-subnetwork PINN architecture (MultiSubPINN) is proposed to address the limitations of traditional PINNs. This is because traditional PINNs used Fully-connected Neural Network (FNN) as main bone. It has problems like different outputs must share parameters within the network structure. However, this parameter sharing nature might lead to potential errors and decrease the model performance. The main architecture of this network is consisted of multiple separated sub-networks and the loss function of each sub-network is calculated separately to update the parameters in each sub-network. By doing this, we can ensure that the optimization for one output doesn't negatively impact others, therefore providing a more flexible, scalable, and potentially more accurate alternative for the evaluation. Besides, using randomly distributed training points in the computational domain for PINNs is a common way. However, many physical problems exhibit non-uniform behavior, with certain regions of the domain having more complex dynamics or sharper gradients than others. Randomly distributed training points might not adequately capture these regions, leading to a model that misrepresents the underlying physics in critical areas. We proposed a Physical-based sampling strategy to optimize the training of the PINNs. This study is presented in Chapter 4. In summary, this dissertation has established a new platform that is enabled by interpretable NN to accurately quantify cement paste’s rheological properties and simulate the flow behavior of 3D concrete printing process. The findings suggest that PINNs can not only improve the efficiency and accuracy of simulations but also provide a more profound understanding of the material behaviors and process dynamics critical to 3DCP. The novel PINN structure developed from this dissertation is a powerful general platform that could be applicable to a variety of material domains for complex rheological behavior predictions.

In recent engineering applications using deep learning, physics-informed neural network (PINN) is a new development as it can exploit the underlying physics of engineering systems. The novelty of PINN lies in the use of partial differential equations (PDE) for the loss function. Most PINNs are implemented using automatic differentiation (AD) for training the PDE loss functions. A lesser well-known study is the use of finite difference method (FDM) as an alternative. Unlike an AD based PINN, an immediate benefit of using a FDM based PINN is low implementation cost. In this paper, we propose the use of finite difference method for estimating the PDE loss functions in PINN. Our work is inspired by computational analysis in electromagnetic systems that traditionally solve Laplace’s equation using successive over-relaxation. In the case of Laplace’s equation, our PINN approach can be seen as taking the Laplacian filter response of the neural network output as the loss function. Thus, the implementation of PINN can be very simple. In our experiments, we tested PINN on Laplace’s equation and Burger’s equation. We showed that using FDM, PINN consistently outperforms non-PINN based deep learning. When comparing to AD based PINNs, we showed that our method is faster to compute as well as on par in terms of error reduction.

Predicting positon solutions of a family of nonlinear Schrödinger equations through deep learning algorithm

Partial differential equations (PDEs) are core tools for characterizing physical laws and are widely used in fluid mechanics, power systems, additive manufacturing, and other fields. However, traditional numerical methods are limited by mesh partitioning, leading to a trade-off between accuracy and efficiency in complex geometric domains and high-dimensional problems. While Physically Informed Neural Networks (PINNs) achieve meshless PDE solutions by embedding prior physical knowledge, they still face challenges such as insufficient accuracy, low training efficiency, and poor stability. This study aims to address the core bottlenecks of PINN in solving PDEs by proposing a systematic optimization strategy to improve its numerical solution accuracy, efficiency, and stability. Methodologically, firstly, a composite loss function containing PDE residuals and boundary initial condition constraints is constructed based on automatic differentiation techniques to ensure that the network satisfies physical laws. Secondly, the PINN training system is optimized, including using Neural Architecture Search (NAS-PINN) to automatically match the optimal network structure, designing an adaptive data sampling strategy to improve data utilization, and introducing parallel computing and hardware acceleration techniques to reduce training time. Finally, the effectiveness of the strategy is verified through classical PDEs and engineering problems. The results show that the optimized PINN achieves high-precision numerical approximation in classical PDE solutions and can efficiently solve forward design and backward parameter inversion problems in engineering scenarios. Compared with traditional methods, it exhibits stronger adaptability in complex geometric domains and high-dimensional problems, while improving training efficiency by more than 30% and significantly enhancing stability. This research not only enriches the optimization theory of PINN and provides an efficient new path for PDE numerical solutions, but also promotes the practical application of PINN in engineering fields such as fluid mechanics, additive manufacturing, and power systems, possessing significant theoretical value and application significance.

1-D coupled surface flow and transport equations revisited via the physics-informed neural network approach

An analysis was made of physics-informed neural networks used to solve partial differential equations. The prospects for the implementation of physics-informed neural networks in the MATLAB system are shown. An algorithm for solving partial differential equations in MATLAB using physics-informed neural networks has been developed. On the example of a model problem described by the Poisson equation, physics-informed neural networks were implemented and studied, which showed that MATLAB can be successfully used to implement such networks. MATLAB made it possible to solve the Poisson equation up to the mean square value of the loss function equal to 0.01. The best results were obtained by networks with a small number of layers (3–4) and a sufficiently large number of neurons in each layer (50–100). Comparison with known results showed that MATLAB was inferior to TensorFlow in terms of learning speed. The application of the Optimization Toolbox MATLAB for the implementation of the L-BFGS quasi-Newtonian learning algorithm for physics-informed neural networks was studied. The quasi-Newtonian algorithm makes it possible to increase the accuracy of solving the problem, but requires a lot of training time. As further research, it is recommended to expand the capabilities of the Deep Learning Toolbox by including quasi-Newtonian learning algorithms, in particular, the Levenberg-Marquard algorithm, and new neural network architectures, for example, networks of radial basis functions.

In recent years, Physics-Informed Neural Networks (PINNs) have introduced a novel approach to solving partial differential equations (PDEs) using deep learning techniques. Despite the promising results and rapid advancements in the field, there is a lack of more accurate studies concerning properties related to modeling choices within the deep learning framework, especially, but not exclusively, in solid mechanics problems. In this study, we aim to explore whether the neural network architecture, activation function, optimizer, and sampling method can influence the accuracy of results and training speed for solid mechanics problems. We will focus on elasticity and hyperelasticity problems in 1D, 2D, and 3D dimensions to lay the groundwork for more complex investigations.

Summary Classical reservoir engineering studies require building geological models and solving complex fluid flow transport equations that require high-quality data, numerous computational resources, time and workflows. For large and mature fields data-driven models can be used to get faster answer and to perform production analysis more efficiently. Capacitance Resistive Models (CRM) are a class of methods based on material balance that can be used to estimate production wells liquid rates as a function of injected water and Bottom Hole Pressure (BHP) variations. CRM methods quantify the connectivity between producers and injectors using only dynamic data. An important drawback of CRM is that they can suffer from parameter identification problems. Moreover, the analytical solution can be only obtained in specific conditions: linear variations of BHP and fixed injection rate between two consecutive time steps. In this work we present a new approach combining CRM material balance equations with neural networks in order to obtain more robust and reliable estimation of the CRM parameters (i.e. well connectivity, productivity indices and time constants). This proposal is also interesting since it is not based on any assumption on BHP and injection rates. To this end, we use a recent approach called Physics Informed Neural Networks (PINNs). In this approach neural networks are trained on observed data with additional physics constraints traduced in appropriate loss functions. The parameters of this physical equation are evaluated at the same time as the neural network weights. The introduction of PINNs in our approach raised after testing classical machine learning (ML) models (SVMs, Random Forests …) and deep learning models (MLP, LSTM, RNNs…). Indeed, such models can perform well in some specific cases but usually struggle to produce robust results (i.e. forecasting) in the long term. Unfortunately, such systems do not natively integrate physics constraints. Our aim is to impose physic constraints in neural networks. Thus, we may obtain more stable and reliable results. On the same time, we should be able to account for more behaviors that are not explained by simplified physic equations such as material balance. We performed a full comparison between our approach using PINNs, other standard ML and DL approaches and a given framework of CRMs on two data-sets: a simple but realistic model build using a commercial reservoir simulator, and a real data set. We show that our approach gives more robust results (in terms of MSE) while not suffering from parameter identification issue.

Adaptive transfer learning for PINN

Identifying optimal architectures of physics-informed neural networks by evolutionary strategy

We devise a deep learning solver inspired by physics-informed neural networks (PINNs) to tackle the polymer self-consistent field theory (SCFT) equationsfor one-dimensional AB-diblock copolymers. The PINNs framework comprises two parallel feedforward neural networks that separately represent the segmental partition functions and self-consistent chemical potential fields. The two networks are coupled through a loss function incorporating the governing equation, initial and boundary conditions, and the incompressibility constraint. To avoid the metastable homogeneous solution, the network parameters are initialized based on known self-consistent fields obtained from the numerical pseudospectral method. For copolymers of length N at a given volume fraction of A block (f) and the reduced Flory-Huggins interaction parameter (χN), the minimization of the loss function leads to the converged network parameters that successfully capture the stable lamellar phase. The periodicity of the lamellar structure is correctly reproduced for the explored sets of [f,χN], irrespective of the presumed computational domain size for initialization. Moreover, the proposed PINNs are applicable to the inverse discovery of the interaction parameter and the embedded chemical potential fields for an observed structure. This capability of solving the inverse SCFT problem demonstrates the potential of using PINNs to accelerate the exploration of new polymeric materials.

This paper investigates the estimation of R-values from transient time series using regression outputs of different neural network architectures, specifically a naive multilayer perceptron (MLP) and a physics-informed neural network (PINN). The PINN incorporates a bias correction for steady-state R-value calculations. The ground truth is derived from transient simulations of wall assemblies. A parameter study examines the prediction accuracy of various MLP architectures within the training process. Further evaluations assess the MLP performances on both validation and independent test datasets. The results indicate that neural networks can effectively enhance R-value estimation. However, in the proposed setup, PINNs did not yield significantly better predictions than conventional MLPs. Future research should explore alternative PINN architectures incorporating more temporal context of the transient input data. Additional studies should also investigate the applicability of more context-aware neural networks, such as convolutional or recurrent neural networks.

The generalization of Physics-Informed Neural Networks (PINNs) used to solve the inhomogeneous Helmholtz equation in a simplified three-dimensional room is investigated. PINNs are appealing since they can efficiently integrate a partial differential equation and experimental data by minimizing a loss function. However, a previous study experienced limitations in acoustics regarding the source term. A challenging but realistic excitation case is a confined (e.g., single-point) excitation area, yielding a smooth spatial wave field periodically with the wavelength. Compared to studies using smooth (unrealistic) sound excitation, the network’s generalization capabilities regarding a realistic sound excitation are addressed. Different methods like hyperparameter optimization, adaptive refinement, Fourier feature engineering, and locally adaptive activation functions with slope recovery are tested to tailor the PINN’s accuracy to an experimentally validated finite element analysis reference solution computed with openCFS. The hyperparameter study and optimization are conducted regarding the network depth and width, the learning rate, the used activation functions, and the deep learning backends (PyTorch 2.5.1, TensorFlow 2.18.0 1, TensorFlow 2.18.0 2, JAX 0.4.39). A modified (feature-engineered) PINN architecture was designed using input feature engineering to include the dispersion relation of the wave in the neural network. For smoothly (unrealistic) distributed sources, it was shown that the standard PINNs and the feature-engineered PINN converge to the analytic solution, with a relative error of 0.28% and 2×10−4%, respectively. The locally adaptive activation functions with the slope lead to a relative error of 0.086% with a source sharpness of s=1 m. Similar relative errors were obtained for the case s=0.2 m using adaptive refinement. The feature-engineered PINN significantly outperformed the results of previous studies regarding accuracy. Furthermore, the trainable parameters were reduced to a fraction by Bayesian hyperparameter optimization (around 5%), and likewise, the training time (around 3%) was reduced compared to the standard PINN formulation. By narrowing this excitation towards a single point, the convergence rate and minimum errors obtained of all presented network architectures increased. The feature-engineered architecture yielded a one order of magnitude lower accuracy of 0.20% compared to 0.019% of the standard PINN formulation with a source sharpness of s=1 m. It outperformed the finite element analysis and the standard PINN in terms time needed to obtain the solution, needing 15 min and 30 s on an AMD Ryzen 7 Pro 8840HS CPU (AMD, Santa Clara, CA, USA) for the FEM, compared to about 20 min (standard PINN) and just under a minute of the feature-engineered PINN, both trained on a Tesla T4 GPU (NVIDIA, Santa Clara, CA, USA).