One-Dimensional Radiative Transfer Equation Solution by Physics-Informed Neural Networks: Performance Analysis

Physics-informed neural networks (PINNs) have been widely used to solve various scientific computing problems. However, large training costs limit PINNs for some real-time applications. Although some works have been proposed to improve the training efficiency of PINNs, few consider the influence of initialization. To this end, we propose a New Reptile initialization based Physics-Informed Neural Network (NRPINN). The original Reptile algorithm is a meta-learning initialization method based on labeled data. PINNs can be trained with less labeled data or even without any labeled data by adding partial differential equations (PDEs) as a penalty term into the loss function. Inspired by this idea, we propose the new Reptile initialization to sample more tasks from the parameterized PDEs and adapt the penalty term of the loss. The new Reptile initialization can acquire initialization parameters from related tasks by supervised, unsupervised, and semi-supervised learning. Then, PINNs with initialization parameters can efficiently solve PDEs. Besides, the new Reptile initialization can also be used for the variants of PINNs. Finally, we demonstrate and verify the NRPINN considering both forward problems, including solving Poisson, Burgers, and Schr\"odinger equations, as well as inverse problems, where unknown parameters in the PDEs are estimated. Experimental results show that the NRPINN training is much faster and achieves higher accuracy than PINNs with other initialization methods.

A physics-informed neural network (PINN) uses physics-augmented loss functions, e.g., incorporating the residual term from governing partial differential equations (PDEs), to ensure its output is consistent with fundamental physics laws. However, it turns out to be difficult to train an accurate PINN model for many problems in practice. In this article, we present a novel perspective of the merits of learning in sinusoidal spaces with PINNs. By analyzing behavior at model initialization, we first show that a PINN of increasing expressiveness induces an initial bias around <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">flat output functions</i> . Notably, this initial solution can be very close to satisfying many physics PDEs, i.e., falling into a local minimum of the PINN loss that only minimizes PDE residuals, while still being far from the true solution that jointly minimizes PDE residuals and the initial and/or boundary conditions. It is difficult for gradient descent optimization to escape from such a local minimum trap, often causing the training to stall. We then prove that the sinusoidal mapping of inputs—in an architecture we label as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sf</i> -PINN—is effective to increase input gradient variability, thus avoiding being trapped in such deceptive local minimum. The level of variability can be effectively modulated to match high-frequency patterns in the problem at hand. A key facet of this article is the comprehensive empirical study that demonstrates the efficacy of learning in sinusoidal spaces with PINNs for a wide range of forward and inverse modeling problems spanning multiple physics domains.

Physics-informed neural networks for high-speed flows

Identifying optimal architectures of physics-informed neural networks by evolutionary strategy

In recent years, successful applications of deep learning (DL) have inspired scientists to explore the possibilities of applying DL approaches to modeling scientific problems. Existing studies have revealed that to bake the physics into the DL makes a good supplement to the traditional numerical methods (e.g., finite element, finite volume method) which primarily rely on partial differential equations (PDEs). While DL models are ordinarily trained in a purely data-driven manner, integrating physics into them for simulating scientific problems has several benefits such as (i) physics constraints could regularize the over-parameterized model and hence mitigate the overfitting issue commonly seen in DL; (ii) physics information could also effectively reduce the amount of data needed for training the model; (iii) the resultant physics-informed DL models feature better interpretability and generalizability compared with the conventional black-box model. Furthermore, the powerful expressiveness of the deep network, guaranteed by the universal approximation theorem, makes it a suitable approximator for the solution to a physical system. In this dissertation, we develop two different DL architectures (or approaches), one being continuous scheme-based while the other discrete scheme-based, that leverage physics knowledge for modeling scientific problems. Through comprehensive numerical experiments, we demonstrate the proposed models can be used in solving general PDEs, establishing predictive data-driven models for dynamical systems, identifying the parameters in governing PDEs or even discovering the entire governing PDEs of dynamical systems from scarce and noisy measurements. The continuous model roots on the physics-informed neural network (PINN) which uses a fully connected neural network (FCNN) to approximate the physical fields of a system globally. This model is mesh-free as the residual of the physics (e.g., PDEs, initial/boundary values) is evaluated on a set of collocation points within the physical domain. Several applications including the forward simulations, data-driven simulations and solving inverse problems are presented to exemplify the advantages of PINN over traditional numerical methods. However, the original PINN suffers from inaccurate initial/boundary values due to the weak enforcement of the initial/boundary conditions (I/BCs). To overcome this issue, we propose an improved PINN model by utilizing multiple deep neural networks (DNNs) to construct the solution. Through a DNN pre-trained to represent the initial/boundary values, the approximated solution would obey the given I/BCs forcibly. With several numerical examples, we show that the improved PINN is characterized with much better accuracy on the I/BCs. Though the PINN shows great promise in data-driven modeling and solving inverse problems, some inherent limitations of PINN still exist, such as (i) the solution might lacks fine-scale details due to the global approximation of FCNN; (ii) high computational expense caused by the FCNN it roots on; (iii) incapability to incorporate existing PDE terms (e.g., $\Delta u$) into the network architecture. To overcome these drawbacks, this dissertation also proposes a discrete model - Physics-encoded Recurrent Convolutional Neural Network (PeRCNN) which recurrently updates the solution (or state variable) for time marching. Specifically, it utilizes convolutional neural network (CNN) to capture the spatial patterns of the solution while the recurrent network mimics the forward Euler scheme (or Runge-Kutta scheme) in numerical methods. PeRCNN is a mesh-based and discrete model due to the discretization in time and spatial dimension. The local connectivity of CNN makes PeRCNN more computationally efficient. In addition, the coercive encoding mechanism of physics in PeRCNN, fundamentally different from the PINN relying on soft penalty, ensures the network to rigorously obey given physics. The proposed PeRCNN is successfully applied to solving general PDEs, the data-driven modeling of dynamical systems and the data-driven discovery of governing PDEs from scarce and noisy measurements. Comparisons with the state-of-the-art DL models demonstrate that the proposed PeRCNN possesses excellent computational efficiency, accuracy and generalizability. --Author's abstract

Partial differential equations (PDEs) are core tools for characterizing physical laws and are widely used in fluid mechanics, power systems, additive manufacturing, and other fields. However, traditional numerical methods are limited by mesh partitioning, leading to a trade-off between accuracy and efficiency in complex geometric domains and high-dimensional problems. While Physically Informed Neural Networks (PINNs) achieve meshless PDE solutions by embedding prior physical knowledge, they still face challenges such as insufficient accuracy, low training efficiency, and poor stability. This study aims to address the core bottlenecks of PINN in solving PDEs by proposing a systematic optimization strategy to improve its numerical solution accuracy, efficiency, and stability. Methodologically, firstly, a composite loss function containing PDE residuals and boundary initial condition constraints is constructed based on automatic differentiation techniques to ensure that the network satisfies physical laws. Secondly, the PINN training system is optimized, including using Neural Architecture Search (NAS-PINN) to automatically match the optimal network structure, designing an adaptive data sampling strategy to improve data utilization, and introducing parallel computing and hardware acceleration techniques to reduce training time. Finally, the effectiveness of the strategy is verified through classical PDEs and engineering problems. The results show that the optimized PINN achieves high-precision numerical approximation in classical PDE solutions and can efficiently solve forward design and backward parameter inversion problems in engineering scenarios. Compared with traditional methods, it exhibits stronger adaptability in complex geometric domains and high-dimensional problems, while improving training efficiency by more than 30% and significantly enhancing stability. This research not only enriches the optimization theory of PINN and provides an efficient new path for PDE numerical solutions, but also promotes the practical application of PINN in engineering fields such as fluid mechanics, additive manufacturing, and power systems, possessing significant theoretical value and application significance.

The curse-of-dimensionality taxes computational resources heavily with exponentially increasing computational cost as the dimension increases. This poses great challenges in solving high-dimensional partial differential equations (PDEs), as Richard E. Bellman first pointed out over 60 years ago. While there has been some recent success in solving numerical PDEs in high dimensions, such computations are prohibitively expensive, and true scaling of general nonlinear PDEs to high dimensions has never been achieved. We develop a new method of scaling up physics-informed neural networks (PINNs) to solve arbitrary high-dimensional PDEs. The new method, called Stochastic Dimension Gradient Descent (SDGD), decomposes a gradient of PDEs’ and PINNs’ residual into pieces corresponding to different dimensions and randomly samples a subset of these dimensional pieces in each iteration of training PINNs. We prove theoretically the convergence and other desired properties of the proposed method. We demonstrate in various diverse tests that the proposed method can solve many notoriously hard high-dimensional PDEs, including the Hamilton–Jacobi-Bellman (HJB) and the Schrödinger equations in tens of thousands of dimensions very fast on a single GPU using the PINNs mesh-free approach. Notably, we solve nonlinear PDEs with nontrivial, anisotropic, and inseparable solutions in less than one hour for 1000 dimensions and in 12 h for 100,000 dimensions on a single GPU using SDGD with PINNs. Since SDGD is a general training methodology of PINNs, it can be applied to any current and future variants of PINNs to scale them up for arbitrary high-dimensional PDEs.

In this work, physics-informed neural networks are applied to incompressible two-phase flow problems. We investigate the forward problem, where the governing equations are solved from initial and boundary conditions, as well as the inverse problem, where continuous velocity and pressure fields are inferred from scattered-time data on the interface position. We employ a volume of fluid approach, i.e. the auxiliary variable here is the volume fraction of the fluids within each phase. For the forward problem, we solve the two-phase Couette and Poiseuille flow. For the inverse problem, three classical test cases for two-phase modeling are investigated: (i) drop in a shear flow, (ii) oscillating drop and (iii) rising bubble. Data of the interface position over time is generated by numerical simulation. An effective way to distribute spatial training points to fit the interface, i.e. the volume fraction field, and the residual points is proposed. Furthermore, we show that appropriate weighting of losses associated with the residual of the partial differential equations is crucial for successful training. The benefit of using adaptive activation functions is evaluated for both the forward and inverse problem.

A novel sequential method to train physics informed neural networks for Allen Cahn and Cahn Hilliard equations

In recent engineering applications using deep learning, physics-informed neural network (PINN) is a new development as it can exploit the underlying physics of engineering systems. The novelty of PINN lies in the use of partial differential equations (PDE) for the loss function. Most PINNs are implemented using automatic differentiation (AD) for training the PDE loss functions. A lesser well-known study is the use of finite difference method (FDM) as an alternative. Unlike an AD based PINN, an immediate benefit of using a FDM based PINN is low implementation cost. In this paper, we propose the use of finite difference method for estimating the PDE loss functions in PINN. Our work is inspired by computational analysis in electromagnetic systems that traditionally solve Laplace’s equation using successive over-relaxation. In the case of Laplace’s equation, our PINN approach can be seen as taking the Laplacian filter response of the neural network output as the loss function. Thus, the implementation of PINN can be very simple. In our experiments, we tested PINN on Laplace’s equation and Burger’s equation. We showed that using FDM, PINN consistently outperforms non-PINN based deep learning. When comparing to AD based PINNs, we showed that our method is faster to compute as well as on par in terms of error reduction.

Deep neural network methods for solving forward and inverse problems of time fractional diffusion equations with conformable derivative

3D concrete printing (3DCP) is an innovative construction method that extrudes cementitious materials layer-by-layer to fabricate building components based on a digital model. 3DCP has gained increasing adoption globally for projects like buildings, bridges, retaining walls, and stormwater management systems. However, 3DCP places stringent demands on the rheological properties of the printable cementitious materials. Rheology is critical for the flow and buildability of fresh concrete during extrusion. The mixture must have a low yield stress for pumping, moderate viscosity to hold its shape after deposition, minimal bleeding and segregation of aggregates, as well as responsive rheology that can transition from fluid to solid state as layers are printed sequentially. In this dissertation, we aim to solve the challenges mentioned above including: (1) accurate and efficient quantification of rheological and thixotropic properties of cementitious materials, (2) evaluation of flow behavior of fresh concrete during extrusion. Using traditional experimental methods to evaluate rheological properties of cement paste is labor-intensive and time-consuming. Numerical simulations like Finite Element Method (FEM) or Computational Fluid Dynamics (CFD) can help increase the speed of finding the most suitable material for 3DCP. However, these simulation methods are costly in computing power, making them less desirable for simulating the dynamic and complex process of 3DCP. Moreover, mesh-based approaches are not suitable for complex geometries as the mesh generation would be challenging with curved boundaries, small features, holes or thin regions. Physics-Informed Neural Network (PINN) is introduced in the work as an innovative solution to simulate and model the whole process of 3DCP. It can leverage the flexibility and computational efficiency of data-based neural networks, which can accelerate the learning and convergence speed. At the same time, Partial Differential Equations (PDEs) would be embedded in the network structure to increase the accuracy of its prediction. The integrated governing physical laws can help the neural network understanding the underlying physics of the 3DCP process, potentially increasing the accuracy and reliability of simulations. A significant part of the research is dedicated to understanding the rheological and thixotropic behavior of cementitious materials, which is critical for the material design in 3DCP. We developed a Rheology-informed Neural Network, RheologyNet, where we embedded the rheological constitutive laws into the loss function. These physical laws can help the network focus on simulating the rheological properties of cementitious materials. Moreover, a thixotropic evaluation system has been established based on the proposed model to evaluate the thixotropic property of cement paste. This study is presented in Chapter 2. Then, we proposed a Navier-Stokes Informed Neural Network (NSINN) to study the flow behavior of cement paste in the 3DCP barrel and nozzle since it involves complicated Multiscale or Multiphysics behaviors, such as anisotropy and non-uniform shear rate distribution. The Navier Stokes Equation and the rheological constitutive equations are coupled and embedded into the NSINN architecture to simulate the flow behavior in the barrel during 3D-printing extrusion process. This approach demonstrates the potential of PINNs to provide high-accuracy simulations of velocity and pressure fields in a computationally efficient manner compared to traditional mesh-based models. Also, we used the NSINN to learn the relationship between the nozzle size and the printing quality. The details of the study are presented in Chapter 3. At last, a multi-subnetwork PINN architecture (MultiSubPINN) is proposed to address the limitations of traditional PINNs. This is because traditional PINNs used Fully-connected Neural Network (FNN) as main bone. It has problems like different outputs must share parameters within the network structure. However, this parameter sharing nature might lead to potential errors and decrease the model performance. The main architecture of this network is consisted of multiple separated sub-networks and the loss function of each sub-network is calculated separately to update the parameters in each sub-network. By doing this, we can ensure that the optimization for one output doesn't negatively impact others, therefore providing a more flexible, scalable, and potentially more accurate alternative for the evaluation. Besides, using randomly distributed training points in the computational domain for PINNs is a common way. However, many physical problems exhibit non-uniform behavior, with certain regions of the domain having more complex dynamics or sharper gradients than others. Randomly distributed training points might not adequately capture these regions, leading to a model that misrepresents the underlying physics in critical areas. We proposed a Physical-based sampling strategy to optimize the training of the PINNs. This study is presented in Chapter 4. In summary, this dissertation has established a new platform that is enabled by interpretable NN to accurately quantify cement paste’s rheological properties and simulate the flow behavior of 3D concrete printing process. The findings suggest that PINNs can not only improve the efficiency and accuracy of simulations but also provide a more profound understanding of the material behaviors and process dynamics critical to 3DCP. The novel PINN structure developed from this dissertation is a powerful general platform that could be applicable to a variety of material domains for complex rheological behavior predictions.

Physics-informed neural networks (PINN) have recently become attractive for solving partial differential equations (PDEs) that describe physics laws. By including PDE-based loss functions, physics laws such as mass balance are enforced softly in PINN. This paper investigates how mass balance constraints are satisfied when PINN is used to solve the resulting PDEs. We investigate PINN’s ability to solve the 1D saturated groundwater flow equations (diffusion equations) for homogeneous and heterogeneous media and evaluate the local and global mass balance errors. We compare the obtained PINN’s solution and associated mass balance errors against a two-point finite volume numerical method and the corresponding analytical solution. We also evaluate the accuracy of PINN in solving the 1D saturated groundwater flow equation with and without incorporating hydraulic heads as training data. We demonstrate that PINN’s local and global mass balance errors are significant compared to the finite volume approach. Tuning the PINN’s hyperparameters, such as the number of collocation points, training data, hidden layers, nodes, epochs, and learning rate, did not improve the solution accuracy or the mass balance errors compared to the finite volume solution. Mass balance errors could considerably challenge the utility of PINN in applications where ensuring compliance with physical and mathematical properties is crucial.

Physics-informed neural networks (PINNs) have made significant strides in modeling dynamical systems governed by partial differential equations (PDEs). However, their generalization capabilities across varying scenarios remain limited. To overcome this limitation, we propose physics-informed dynamics representation learner (PiDo), a novel physics-informed neural PDE solver designed to generalize effectively across diverse PDE configurations, including varying initial conditions, PDE coefficients, and training-time horizons. PiDo exploits the shared underlying structure of dynamical systems with different properties by projecting PDE solutions into a latent space using auto-decoding. It then learns the dynamics of these latent representations, conditioned on the PDE coefficients. Despite its promise, integrating latent dynamics models within a physics-informed framework poses challenges due to the optimization difficulties associated with physics-informed losses. To address these challenges, we introduce a novel approach that diagnoses and mitigates these issues within the latent space. This strategy employs straightforward yet effective regularization techniques, enhancing both the temporal extrapolation performance and the training stability of PiDo. We validate PiDo on a range of benchmarks, including 1-D combined equations and 2-D Navier-Stokes equations. In addition, we demonstrate the transferability of its learned representations to downstream applications such as long-term integration and inverse problems.

In this paper, we propose an extrinsic approach based on physics-informed neural networks (PINNs) for solving the partial differential equations (PDEs) on surfaces embedded in high dimensional space. PINNs are one of the deep learning-based techniques. Based on the training data and physical models, PINNs introduce the standard feedforward neural networks (NNs) to approximate the solutions to the PDE systems. Using automatic differentiation, the PDEs information could be encoded into NNs and a loss function. To deal with the surface differential operators in the loss function, we combine the extrinsic approach with PINNs and then express that loss function in extrinsic form. Subsequently, the loss function could be minimized extrinsically with respect to the NN parameters. Numerical results demonstrate that the extrinsic approach based on PINNs for surface problems has good accuracy and higher efficiency compared with the embedding approach based on PINNs. In addition, the strong nonlinear mapping ability of NNs makes this approach robust in solving time-dependent nonlinear problems on more complex surfaces.