Extending the Universal Approximation Theorem for a Broad Class of Hypercomplex-Valued Neural Networks

Given N data pairs {X i , y i }, i=1,2,...,N, where each X i is an n-dimensional vector of independent variables \((X_i = )\) and y i is a dependent variable, the function approximation problem (FAP) is finding a function that best explains the N pairs of X i and y i . From the universal approximation theorem and inherent approximation capabilities proved by various researches, artificial neural networks (ANNs) are considered as powerful function approximators. There are two main issues on the feedforward neural networks’ performance. One is to determine its structure. The other issue is to specify the weights of a network that minimizes its error. Genetic algorithm (GA) is a global search technique and is useful for complex optimization problems. So, it has been considered to have potential to reinforce the performance of neural networks. Many researchers tried to optimize the weights of networks using genetic algorithms alone or combined with the backpropagation algorithm. Others also tried to find a good topology that is even more difficult and called a “black art.”

In this chapter, we derive neural networks from the viewpoint of physical models. A neural network is a nonlinear function that maps an input to an output, and giving the network is equivalent to giving a function called an error function in the case of supervised learning. By considering the output as dynamical degrees of freedom and the input as an external field, various neural networks and their deepened versions are born from simple Hamiltonians. Training (learning) is a procedure for reducing the value of the error function, and we will learn the specific method of backpropagation using the bra-ket notation popular in quantum mechanics. And we will look at how the “universal approximation theorem” works, which is why neural networks can express connections between various types of data.

In recent years, successful applications of deep learning (DL) have inspired scientists to explore the possibilities of applying DL approaches to modeling scientific problems. Existing studies have revealed that to bake the physics into the DL makes a good supplement to the traditional numerical methods (e.g., finite element, finite volume method) which primarily rely on partial differential equations (PDEs). While DL models are ordinarily trained in a purely data-driven manner, integrating physics into them for simulating scientific problems has several benefits such as (i) physics constraints could regularize the over-parameterized model and hence mitigate the overfitting issue commonly seen in DL; (ii) physics information could also effectively reduce the amount of data needed for training the model; (iii) the resultant physics-informed DL models feature better interpretability and generalizability compared with the conventional black-box model. Furthermore, the powerful expressiveness of the deep network, guaranteed by the universal approximation theorem, makes it a suitable approximator for the solution to a physical system. In this dissertation, we develop two different DL architectures (or approaches), one being continuous scheme-based while the other discrete scheme-based, that leverage physics knowledge for modeling scientific problems. Through comprehensive numerical experiments, we demonstrate the proposed models can be used in solving general PDEs, establishing predictive data-driven models for dynamical systems, identifying the parameters in governing PDEs or even discovering the entire governing PDEs of dynamical systems from scarce and noisy measurements. The continuous model roots on the physics-informed neural network (PINN) which uses a fully connected neural network (FCNN) to approximate the physical fields of a system globally. This model is mesh-free as the residual of the physics (e.g., PDEs, initial/boundary values) is evaluated on a set of collocation points within the physical domain. Several applications including the forward simulations, data-driven simulations and solving inverse problems are presented to exemplify the advantages of PINN over traditional numerical methods. However, the original PINN suffers from inaccurate initial/boundary values due to the weak enforcement of the initial/boundary conditions (I/BCs). To overcome this issue, we propose an improved PINN model by utilizing multiple deep neural networks (DNNs) to construct the solution. Through a DNN pre-trained to represent the initial/boundary values, the approximated solution would obey the given I/BCs forcibly. With several numerical examples, we show that the improved PINN is characterized with much better accuracy on the I/BCs. Though the PINN shows great promise in data-driven modeling and solving inverse problems, some inherent limitations of PINN still exist, such as (i) the solution might lacks fine-scale details due to the global approximation of FCNN; (ii) high computational expense caused by the FCNN it roots on; (iii) incapability to incorporate existing PDE terms (e.g., $\Delta u$) into the network architecture. To overcome these drawbacks, this dissertation also proposes a discrete model - Physics-encoded Recurrent Convolutional Neural Network (PeRCNN) which recurrently updates the solution (or state variable) for time marching. Specifically, it utilizes convolutional neural network (CNN) to capture the spatial patterns of the solution while the recurrent network mimics the forward Euler scheme (or Runge-Kutta scheme) in numerical methods. PeRCNN is a mesh-based and discrete model due to the discretization in time and spatial dimension. The local connectivity of CNN makes PeRCNN more computationally efficient. In addition, the coercive encoding mechanism of physics in PeRCNN, fundamentally different from the PINN relying on soft penalty, ensures the network to rigorously obey given physics. The proposed PeRCNN is successfully applied to solving general PDEs, the data-driven modeling of dynamical systems and the data-driven discovery of governing PDEs from scarce and noisy measurements. Comparisons with the state-of-the-art DL models demonstrate that the proposed PeRCNN possesses excellent computational efficiency, accuracy and generalizability. --Author's abstract

LU decomposition and Toeplitz decomposition of a neural network

To develop, train and test neural networks for predicting heart surface potentials (HSPs) from body surface potentials (BSPs). The method re-frames traditional inverse problems of electrocardiography into regression problems, constraining the solution space by decomposing signals with multidimensional Gaussian impulse basis functions. Impulse HSPs were generated with single Gaussian basis functions at discrete heart surface locations and projected to corresponding BSPs using a volume conductor torso model. Both BSP (inputs) and HSP (outputs) were mapped to regular 2D surface meshes and used to train a neural network. Predictive capabilities of the network were tested with unseen synthetic and experimental data. A dense full connected single hidden layer neural network was trained to map body surface impulses to heart surface Gaussian basis functions for reconstructing HSP. Synthetic pulses moving across the heart surface were predicted from the neural network with root mean squared error of 9.1±1.4%. Predicted signals were robust to noise up to 20 dB and errors due to displacement and rotation of the heart within the torso were bounded and predictable. A shift of the heart 40 mm toward the spine resulted in a 4% increase in signal feature localization error. The set of training impulse function data could be reduced, and prediction error remained bounded. Recorded HSPs from in-vitro pig hearts were reliably decomposed using space-time Gaussian basis functions. Activation times calculated from predicted HSPs for left-ventricular pacing had a mean absolute error of 10.4±11.4 ms. Other pacing scenarios were analyzed with similar success. Impulses from Gaussian basis functions are potentially an effective and robust way to train simple neural network data models for reconstructing HSPs from decomposed BSPs. The HSPs predicted by the neural network can be used to generate activation maps that non-invasively identify features of cardiac electrical dysfunction and can guide subsequent treatment options.

Function approximation is one of the core tasks that are solved using neural networks in the context of many engineering problems. However, good approximation results need good sampling of the data space, which usually requires exponentially increasing volume of data as the dimensionality of the data increases. At the same time, often the high-dimensional data is arranged around a much lower dimensional manifold. Here we propose the breaking of the function approximation task for high-dimensional data into two steps: (1) the mapping of the high-dimensional data onto a lower dimensional space corresponding to the manifold on which the data resides and (2) the approximation of the function using the mapped lower dimensional data. We use over-complete self-organizing maps (SOMs) for the mapping through unsupervised learning, and single hidden layer neural networks for the function approximation through supervised learning. We also extend the two-step procedure by considering support vector machines and Bayesian SOMs for the determination of the best parameters for the nonlinear neurons in the hidden layer of the neural networks used for the function approximation. We compare the approximation performance of the proposed neural networks using a set of functions and show that indeed the neural networks using combined unsupervised and supervised learning outperform in most cases the neural networks that learn the function approximation using the original high-dimensional data.

Universal approximation capability is an important research topic in artificial neural networks. The purpose of this study is to investigate universal approximation capability of a single hidden layer feed forward multiple 2π-periodic approximate identity neural networks in two function spaces. We present the notion of multiple 2π-periodic approximate identity. With respect to this notion, we prove two theorems in the space of continuous multivariate 2π-periodic functions. The second theorem shows that the above networks have universal approximation capability. The proof of the theorem uses a technique based on the notion of epsilon-net. Moreover, we discuss the universal approximation capability of the networks in the space of Lebesgue integrable multivariate 2π-periodic functions. The results of this study will be able to extend the standard theory of the universal approximation capability of feedforward neural networks.

One of the main computer-learning tools is an (artificial) neural network (NN); based on the values y (p) of a certain physical quantity y at several points x (p) =(x 1 (p) ,...,x n (p) ), the NN finds a dependence y = f(x1,...,x n ) that explains all known observations and predicts the value of y for other x = (x1,...,xn). The ability to describe an arbitrary dependence follows from the universal approximation theorem, according to which an arbitrary continuous function of a bounded set can be, within a given accuracy, approximated by an appropriate NN.

Stochastic resonance (SR) is a natural process that without limit increases the precision of signal measurements in biological and physical sciences. Most artificial neural networks (NNs) are implemented on digital computers of fixed-precision. A NN accessing universal approximation and a computational complexity class more powerful that of a Turing machine needs analog signals utilizing SR’s limitless precision increase. This paper links an analog recurrent (AR) NN theorem, SR, BPP/log* (a physically realizable, super-Turing computation class), and universal approximation so NNs following them can be made computationally more powerful. An optical neural network mimicking chaos indicates super-Turing computation has been achieved. Additional tests are needed which can verify superTuring computation, show its superiority, and demonstrate its practical benefits. Truly powerful cognitively inspired computation needs to access the combination of ARNNs, SR, super-Turing mathematical complexity, and universal approximation.

This paper presents a novel robust adaptive trajectory linearization control (RATLC) method for a class of uncertain nonlinear systems based on a single hidden layer neural networks disturbance observer (SDO). The term ldquodisturbancerdquo used in this paper refers to the combination of model uncertainties and external disturbances. By utilizing the universal approximation property of neural networks with useful information on the controlled plant, the SDO can monitor time-varying disturbance very well. A robust adaptive term is added to overcome reconstruction error. Conditions are derived which guarantee ultimate boundedness of all the errors in the combined system. Excellent disturbance attenuation ability and strong robustness of the proposed RATLC method are demonstrated by an example of inverted pendulum control problem.

The identification of nonlinear systems can be posed as a nonlinear functional approximation problem. From the Weierstrass Theorem (Powell, 1981) and the Kolmogorov theorem (Sprecher, 1965) in approximation theory, it is shown that the polynomial and many other approximation schemes can approximate a continuous function arbitrarily well. In recent years, a number of nonlinear system identification approaches, particularly identification using neural networks, based on the universal approximation theorem (Cybenko, 1989), are applications of a similar mathematical approach.

<div class="section abstract"><div class="htmlview paragraph">Virtual sensing or estimation of emission species such as NOx at the engine exhaust using the appropriate engine measurables and leveraging it for control/diagnosis is a challenging task given the highly nonlinear and dynamic nature of the combustion process. This article presents development of virtual engine-out NOx (EONOx) sensor using two different supervised dynamic artificial neural network (ANN) topologies, namely, trained recurrent neural network (RNN) and wavelet neural network (WNN). The proposed RNN architecture is a single hidden layer neural network with permutations of feedback connections between the inter- and intra-layer nodes. The RNN resembles a nonlinear state-space model mapping select engine measurables and the engine-out NOx and is trained using a variant of real-time recurrent learning (RTRL) algorithm. The WNN architecture is a single hidden layer neural network comprising hidden layer nodes with wavelets as activation functions. The activation functions of the WNN nodes are adapted for their form and time shift along with their synaptic weights in the supervised learning method. The topologies are validated in virtual environment using modeled data as well as experimental data. Approaches toward leveraging these virtual sensors for better NOx control, both at the engine-out and system-out level are discussed along with their benefits. The limitations of such data-based virtual sensors are stated. The outcome of this work is methodology to select appropriate ANN topology and training it for efficient EONOx virtual sensor and leveraging it for control at engine and tailpipe level.</div></div>

This paper describes the results of a study focused on comparing the performance of three different neural augmentations of the dynamic inversion-based control laws used for fault tolerant purposes on the NASA IFCS F-15 aircraft. The performance of the specific neural algorithms, the extended minimal resource allocating networks algorithm, the single hidden layer neural network, and the SigmaPi neural network have been compared. The comparison has been conducted in terms of specific parameters relative to the tracking of desirable handling qualities following the injection of simulated failures on the actuators of the right stabilator and the left canard of the NASA F-15 aircraft. The simulation results have shown that all three neural networks have promising performance with the extended minimal resource allocating networks algorithm slightly outperforming the other algorithms.

In this paper, a neural network based adaptive state feedback control scheme is proposed to solve the problem of trajectory tracking in the joint space for robotic manipulators with the presence of high uncertainty in the system parameters. First nonlinear behavior of the robot is approximately eliminated by applying a linearizing control, the closed loop dynamics is stabilized using static compensator. Then the parametric uncertainty is parameterized and eliminated using a single hidden layer neural network. The adaptation law for the NN and the boundedness of the error signals are proved from Lyapunov theorem of stability. The performance of the proposed controller is validated on two-link manipulator and demonstrated through computer simulation.

A measure of success for any learning algorithm is how useful it is in a variety of learning situations. Those learning algorithms that support universal function approximation can theoretically be applied to a very large and interesting class of learning problems. Many kinds of neural network architectures have already been shown to support universal approximation. In this paper, we will provide a proof to show that Fuzzy ART augmented with a single layer of perceptrons is a universal approximator. Moreover, the Fuzzy ARTMAP neural network architecture, by itself, will be shown to be a universal approximator.