Deep-time neural networks: An efficient approach for solving high-dimensional PDEs

Aiming at the heat conduction equation, the heat conduction model is a very typical example of the partial differential equation. This paper focuses on the development and preliminary knowledge of the partial differential equation, the establishment of the heat conduction model and the solution of the heat conduction model. The partial differential equation with fractional Laplace operator is a kind of typical fractional partial differential equation, which has important applications in the fields of science and engineering. Fractional Laplace operator is a kind of nonlocal quasi-differential operator, which is the infinitesimal generator of L é vy steady-state process. It is essentially different from the classical Laplace operator, which leads to the disappearance of some classical properties, which generally brings difficulties to the study of this kind of problems. It is very difficult to solve the explicit solution of partial differential equation with fractional Laplace operator. In order to improve the accuracy and speed of solving partial differential equation, a high precision method for solving partial differential equation based on deep learning network is proposed. A multilayer radial basis function neural network is constructed and its structural layers are determined. By introducing subnetworks into each layer of the multilayer radial basis function neural network, a composite multilayer radial basis function neural network is constructed to improve the real function approximation performance and operation accuracy of the multilayer radial basis function neural network. The high-precision compound multi-layer radial basis function neural network is used to solve the partial differential equation. By giving a specific example of solving the partial differential equation, the solution effect of the method is tested. The results show that the method has very high solution accuracy under the four-layer network structure, which can improve about 1.5 orders of magnitude compared with the first layer. Under different number of training samples, it has higher solution accuracy and speed, and the comprehensive performance is superior. Therefore, it is a practical but challenging work to study the existence of solutions of partial differential equations with fractional Laplace operators. This paper mainly describes the research progress and trends of the existence of solutions of several classes of partial differential equations with fractional Laplace operators, including some of the work done by the author in this field in recent years.

Fast and scalable computation of shape-morphing nonlinear solutions with application to evolutional neural networks

Deep neural networks based temporal-difference methods for high-dimensional parabolic partial differential equations

Assessing basic research : Some partial indicators of scientific progress in radio astronomy

Unsupervised anomaly detection in pressurized water reactor digital twins using autoencoder neural networks

This study discusses the negative impact of the derivative of the activation functions in the output layer of artificial neural networks, in particular in continual learning. We propose Hebbian descent as a theoretical framework to overcome this limitation, which is implemented through an alternative loss function for gradient descent we refer to as Hebbian descent loss. This loss is effectively the generalized log-likelihood loss and corresponds to an alternative weight update rule for the output layer wherein the derivative of the activation function is disregarded. We show how this update avoids vanishing error signals during backpropagation in saturated regions of the activation functions, which is particularly helpful in training shallow neural networks and deep neural networks where saturating activation functions are only used in the output layer. In combination with centering, Hebbian descent leads to better continual learning capabilities. It provides a unifying perspective on Hebbian learning, gradient descent, and generalized linear models, for all of which we discuss the advantages and disadvantages. Given activation functions with strictly positive derivative (as often the case in practice), Hebbian descent inherits the convergence properties of regular gradient descent. While established pairings of loss and output layer activation function (e.g., mean squared error with linear or cross-entropy with sigmoid/softmax) are subsumed by Hebbian descent, we provide general insights for designing arbitrary loss activation function combinations that benefit from Hebbian descent. For shallow networks, we show that Hebbian descent outperforms Hebbian learning, has a performance similar to regular gradient descent, and has a much better performance than all other tested update rules in continual learning. In combination with centering, Hebbian descent implements a forgetting mechanism that prevents catastrophic interference notably better than the other tested update rules. When training deep neural networks, our experimental results suggest that Hebbian descent has better or similar performance as gradient descent.

Derivative-informed projected neural networks for high-dimensional parametric maps governed by PDEs

The curse-of-dimensionality taxes computational resources heavily with exponentially increasing computational cost as the dimension increases. This poses great challenges in solving high-dimensional partial differential equations (PDEs), as Richard E. Bellman first pointed out over 60 years ago. While there has been some recent success in solving numerical PDEs in high dimensions, such computations are prohibitively expensive, and true scaling of general nonlinear PDEs to high dimensions has never been achieved. We develop a new method of scaling up physics-informed neural networks (PINNs) to solve arbitrary high-dimensional PDEs. The new method, called Stochastic Dimension Gradient Descent (SDGD), decomposes a gradient of PDEs’ and PINNs’ residual into pieces corresponding to different dimensions and randomly samples a subset of these dimensional pieces in each iteration of training PINNs. We prove theoretically the convergence and other desired properties of the proposed method. We demonstrate in various diverse tests that the proposed method can solve many notoriously hard high-dimensional PDEs, including the Hamilton–Jacobi-Bellman (HJB) and the Schrödinger equations in tens of thousands of dimensions very fast on a single GPU using the PINNs mesh-free approach. Notably, we solve nonlinear PDEs with nontrivial, anisotropic, and inseparable solutions in less than one hour for 1000 dimensions and in 12 h for 100,000 dimensions on a single GPU using SDGD with PINNs. Since SDGD is a general training methodology of PINNs, it can be applied to any current and future variants of PINNs to scale them up for arbitrary high-dimensional PDEs.

Deep Learning (DL) is an area of artificial intelligence (AI), wherein machine learning (ML) using convolutional neural networks to detect patterns in data and forecast its worth. It excels in tough settings like climate change and can handle sophisticated jobs such as image and speech recognition, natural language processing, and others. Deep learning models have found applications in a variety of fields due to their ability to reflect complicated patterns in data. Deep learning applications have the capacity to analyze and automatically classify large volumes of Internet traffic. Deep learning-based solutions that automate the detection of attacks and address complicated cybersecurity issues are gaining popularity. This study extensively describes the promising uses of deep learning, which is based on many layers of artificial neural networks, in a wide range of security challenges. Before critically and comparatively reviewing state-of-the-art solutions from the literature, we will highlight the main properties of typical deep learning architectures used in cybersecurity applications. Deep learning highlights developing concepts and provides an overview of required resources, such as a general framework and appropriate datasets. The limits of the evaluated works are highlighted, as well as a vision of the present challenges in the field, providing useful insights and best practices for researchers and developers working on related problems.

Neural machine learning methods, such as deep neural networks (DNN), have achieved remarkable success in a number of complex data processing tasks. These methods have arguably had their strongest impact on tasks such as image and audio processing - data processing domains in which humans have long held clear advantages over conventional algorithms. In contrast to biological neural systems, which are capable of learning continuously, deep artificial networks have a limited ability for incorporating new information in an already trained network. As a result, methods for continuous learning are potentially highly impactful in enabling the application of deep networks to dynamic data sets. Here, inspired by the process of adult neurogenesis in the hippocampus, we explore the potential for adding new neurons to deep layers of artificial neural networks in order to facilitate their acquisition of novel information while preserving previously trained data representations. Our results on the MNIST handwritten digit dataset and the NIST SD 19 dataset, which includes lower and upper case letters and digits, demonstrate that neurogenesis is well suited for addressing the stability-plasticity dilemma that has long challenged adaptive machine learning algorithms.

In the last decade, the Computational Intelligence tools (CI), including Artificial Neural Networks (ANN) and Fuzzy Systems (FS), applying soft computing, became universal means for many applications. Because of their approximation and learning capabilities, the ANNs have been widely employed to dynamic process modeling, identification, prediction and control, (Boskovic & Narendra, 1995; Haykin, 1999; Bulsari & Palosaari, 1993; Deng & Li, 2003; Deng et al., 2005; Gonzalez-Garcia et al., 1998; Padhi & Balakrishnan, 2003; Padhi et al., 2001; Ray,1989). Many applications have been done for identification and control of biotechnological plants too, (Padhi et al., 2001). Among several possible neural network architectures the ones most widely used are the Feedforward NN (FFNN) and the Recurrent NN (RNN), (Haykin, 1999). The main NN property namely the ability to approximate complex non-linear relationships without prior knowledge of the model structure makes them a very attractive alternative to the classical modeling and control techniques. Also, a great boost has been made in the applied NN-based adaptive control methodology incorporating integral plus state control action in the control law, (Baruch et al., 2004; Baruch & Garrido, 2005; Baruch et al., 2007). The FFNN and the RNN have been applied for Distributed Parameter Systems (DPS) identification and control too. In (Pietil & Koivo, 1996), a RNN is used for system identification and process prediction of a DPS dynamics an adsorption column for wastewater treatment of water contaminated with toxic chemicals. In (Deng & Li, 2003; Deng et al., 2005) a spectral-approximation-based intelligent modeling approach, including NNs for state estimation and system identification, is proposed for the distributed thermal processing of the snap curing oven DPS that is used in semiconductor packaging industry. In (Bulsari & Palosaari, 1993), it is presented a new methodology for the identification of DPS, based on NN architectures, motivated by standard numerical discretization techniques used for the solution of Partial Differential Equation (PDE). In (Padhi & Balakrishnan, 2003), an attempt is made to use the philosophy of the NN adaptivecritic design to the optimal control of distributed parameter systems. In (Padhi et al., 2001) the concept of proper orthogonal decomposition is used for the model reduction of DPS to form a reduced order lumped parameter problem. The optimal control problem is then solved in the time domain, in a state feedback sense, following the philosophy of adaptive critic NNs. The control solution is then mapped back to the spatial domain using the same basis functions. In (Pietil & Koivo, 1996), measurement data of an industrial process are

Artificial neural networks (ANNs) have very successfully been used in numerical simulations for a series of computational problems ranging from image classification/image recognition, speech recognition, time series analysis, game intelligence, and computational advertising to numerical approximations of partial differential equations (PDEs). Such numerical simulations suggest that ANNs have the capacity to very efficiently approximate high-dimensional functions and, especially, indicate that ANNs seem to admit the fundamental power to overcome the curse of dimensionality when approximating the high-dimensional functions appearing in the above named computational problems. There are a series of rigorous mathematical approximation results for ANNs in the scientific literature. Some of them prove convergence without convergence rates and some of these mathematical results even rigorously establish convergence rates but there are only a few special cases where mathematical results can rigorously explain the empirical success of ANNs when approximating high-dimensional functions. The key contribution of this article is to disclose that ANNs can efficiently approximate high-dimensional functions in the case of numerical approximations of Black-Scholes PDEs. More precisely, this work reveals that the number of required parameters of an ANN to approximate the solution of the Black-Scholes PDE grows at most polynomially in both the reciprocal of the prescribed approximation accuracy ε > 0 \varepsilon > 0 and the PDE dimension d ∈ N d \in \mathbb {N} . We thereby prove, for the first time, that ANNs do indeed overcome the curse of dimensionality in the numerical approximation of Black-Scholes PDEs.

This paper explores the solutions of certain fractional partial differential equations using two methods; the first method involves separation of variables, which is a common technique for solving partial differential equations. However, since many equations cannot be separated in this way, the tensor product of Banach spaces method is applied to find the atomic solutions. To solve the resulting ordinary differential equations, the reproducing Kernel Hilbert space method is used to find numerical solutions, which are then used to find the numerical solution of the partial differential equation. The residual errors indicate that this method is effective and powerful. In summary, this paper presents a study on the solutions of certain fractional partial differential equations using two methods and demonstrates the effectiveness of these methods in finding numerical solutions.

The authors demonstrate how the use of arbitrary nonlinearities can improve the storage capacity of a class of layered classification artificial neural networks (L-CANNs). The network's storage capacity is on the order of the number of neurons used to stimulate the response. L-CANNs can be trained by viewing the training data only once. Classification boundaries corresponding to maximum points of confusion, if known, can also be learned. Iteration is not required in the recall mode. The manner in which a network responds to data outside the training set can be straightforwardly evaluated. The L-CANN also has the ability to recognize the unfamiliarity of stimuli for which it was not trained. >

In scientific and engineering calculation, the effective solution of partial differential equations (PDEs) has great significance. This paper presents an innovative method based on the combination of a U-Net neural network with Fourier neural operators, aiming to improve the accuracy and efficiency of solving partial differential equations. U-Net neural networks with a unique encoding–decoder structure and hopping connections can efficiently extract and integrate spatial-domain features and accurately describe the spatial structure of PDEs. The Fourier neural operator combines the advantages of neural networks and Fourier transform to learn the mapping relationship of high-dimensional data by training the linear transformation parameters in the detailed frequency domain of the data. Fourier transform is used to explore and process the frequency-domain features and capture the frequency characteristics of the solution of partial differential equations. By combining the two, this method not only retains spatial details but also makes full use of frequency-domain information, which significantly reduces the dependence of the model on large-scale data and improves the generalization ability of the model. The experimental results show that compared with the traditional method, this method performs well in various complex partial differential equation (PDE) solving tasks and achieves high accuracy.