Representation formulas and pointwise properties for Barron functions

In this paper, we study the compression of a target two-layer neural network with N nodes into a compressed network with M <N nodes. More precisely, we consider the setting in which the weights of the target network are i.i.d. sub-Gaussian, and we minimize the population L <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> loss between the outputs of the target and of the compressed network, under the assumption of Gaussian inputs. By using tools from high-dimensional probability, we show that this non-convex problem can be simplified when the target network is sufficiently over-parameterized, and provide the error rate of this approximation as a function of the input dimension and N. In this mean-field limit, the simplified objective, as well as the optimal weights of the compressed network, does not depend on the realization of the target network, but only on expected scaling factors. Furthermore, for networks with ReLU activation, we conjecture that the optimum of the simplified optimization problem is achieved by taking weights on the Equiangular Tight Frame (ETF), while the scaling of the weights and the orientation of the ETF depend on the parameters of the target network. Numerical evidence is provided to support this conjecture.

A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated. General initialization schemes as well as general regimes for the network width and training data size are considered. In the over-parametrized regime, it is shown that gradient descent dynamics can achieve zero training loss exponentially fast regardless of the quality of the labels. In addition, it is proved that throughout the training process the functions represented by the neural network model are uniformly close to that of a kernel method. For general values of the network width and training data size, sharp estimates of the generalization error is established for target functions in the appropriate reproducing kernel Hilbert space.

A neural network controller which is used for controlling unknown discrete-time DARMA systems is described. In a first stage, a two-layered neural network is used to estimate the unknown plant dynamics. The Widrow-Hoff delta rule is used as the learning algorithm for this network so as to minimize the difference between the plant actual response and that predicted by the neural network. In a second stage, the control law is generated online using a second two-layered neural network so that the plant output is brought to a desired reference signal. Simulation examples are presented to evaluate the design.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher–student setup, where one network, the student, is trained on data generated by another network, called the teacher. We show how the dynamics of stochastic gradient descent (SGD) is captured by a set of differential equations and prove that this description is asymptotically exact in the limit of large inputs. Using this framework, we calculate the final generalisation error of student networks that have more parameters than their teachers. We find that the final generalisation error of the student increases with network size when training only the first layer, but stays constant or even decreases with size when training both layers. We show that these different behaviours have their root in the different solutions SGD finds for different activation functions. Our results indicate that achieving good generalisation in neural networks goes beyond the properties of SGD alone and depends on the interplay of at least the algorithm, the model architecture, and the data set.

The lower and upper bounds for the information capacity of two-layer feedforward neural networks with binary interconnections, integer thresholds for the hidden units, and zero threshold for the output unit is obtained through two steps. First, through a constructive approach based on statistical analysis, it is shown that a specifically constructed (N-2L-1) network with N input units, 2L hidden units, and one output unit is capable of implementing, with almost probability one, any dichotomy of O(W/ln W) random samples drawn from some continuous distributions, where W is the total number of weights of the network. This quantity is then used as a lower bound for the information capacity C of all (N-2L-1) networks with binary weights. Second, an upper bound is obtained and shown to be O(W) by a simple counting argument. Therefore, we have /spl Omega/(W/ln W)/spl les/C/spl les/O(W).

The momentum method has become the workhorse in the deep learning community. To theoretical understand its success, researchers put efforts in demystifying its convergence properties when optimizing neural networks. For the convex problem, it is well-known that the triple momentum (TM) method owns the fastest theoretical convergence rate among all first-order methods. However, there exists no theoretical convergence results about the TM method in solving the non-convex neural networks training problem, let alone its acceleration guarantee. In this paper, we focus on the training process of the TM method for a two-layer ReLU neural network. Inspired by the accurate characterization of the high-resolution dynamical system, we consider the high-resolution ordinary differential equation (ODE) of the TM method. Under the over-parameterized assumption, we derive that the original non-convex optimization problem can be transformed to a strongly convex task. By applying an appropriate Lyapunov function, we prove that the TM method can linearly converge to a global minimum. Compared to the heavy ball method and Nesterov’s accelerated gradient method, our result provides the first guarantee for the acceleration of the TM method in training neural networks. Through empirical experiments, the accelerated convergence of the TM method and the effect of over-parameterization are validated.

After the issue of keywords not effectively representing the theme in topic discovery tasks, this paper proposes a text topic representation method based on Seq2Seq-Attention, aiming to create a concise and refined topic representation. Firstly, the Encoder module uses a two-layer bidirectional recurrent neural network for text information feature extraction, the Decoder sub-module uses a two-layer unidirectional recurrent neural network and combines with an attention mechanism model to complete the decoding task and outputs the desired sequence of popular topic characters. Finally, the effectiveness of the method is verified through comparative experiments.

The key of fast identification algorithm of time-varying modal parameter based on subspace tracking is to find efficient and fast subspace-tracking algorithm.This paper presents a new version of NIC(Novel Information Criterion) using two-layer linear neural network learning for subspace tracking.Comparing with the original algorithm, there is no need to set a key control parameter in advance.Simulation experiments show that new algorithm has a faster convergence in the initial period.

Recently, the sonar signal processing community has seen increased interest in neural networks for nonlinear classification and regression. While innovative neural network architectures have achieved remarkable results in other fields such as computer vision and natural language processing, their application to sonar data presents challenges. The multiple nonlinearities in these networks make it difficult to interpret results. Recent works such as Grad-CAM and Layer-wise Relevance Propagation attempt to empirically visualize what features are being selected by the network, but there has been little theoretical work to explain why neural networks perform so well. However, recent work by Pilanci et al. has shown that two-layer neural networks with a ReLU activation function can be reformulated as a convex optimization that can be solved optimally in polynomial time. Their results rely on the observation that ReLU neural networks are piece-wise linear models that lift the original data into a higher dimensional feature space. Using a valid activation function, we show that this Convex Neural Network framework can be extended to applications where the input data is complex-valued. We evaluate our algorithm on simulated sonar data. [This talk will present research funded by DoD Navy (NEEC) Grant number N001742010016 and the ONR grant numbers N000142112420 and N000142312503.]

In this paper, a Stator Current Model Reference Adaptive System (SC_MRAS) based estimator using neural network for sensorless controlled Six Phase Induction Motor (SPIM) drive is presented. For this scheme the measured stator current components are used as the reference model of the MRAS estimator to avoid the use of a pure integrator and reduce influence of motor parameter variation. The adaptive model of proposal estimator use a two-layer Neural Network (NN) to estimate the stator current, which requires the rotor flux information that can be obtained from the voltage model. A back-propagation learning algorithm is used to minimise the error in current estimation and hence in generating the estimated speed. Speed estimation performance of the proposal MRAS scheme is verified and compared with the classical rotor flux MRAS when applied to sensorless vector control SPIM drive. Simulation results have demonstrated that the proposed MRAS estimator for sensorless control of SPIM drive has good dynamic response, high precision in both the transient and steady modes, the performance of the SC_MRAS using NN is significantly improved especially at low and near zero speed range.

Self-organization of orientation maps in a formal neuron model using a cluster learning rule

The role of cyber security in facing risks and damages is an essential task. The aim of study is first to the effect of a sabotage sample in security on the power market, this is the unavailability of the production or the power outage. This network is a three-bus system consisting of three wind power generation units. This is achieved by running several scenarios in the system. The problem solution approach is considered in minimizing the costs after attacking in the system. The CPLEX solver solves the mixed integer linear programming (MILP) model. Also, the data observed from the output of the GAMS software is used as the input data of the Python software to use data science such as histogram chart analysis and machine learning algorithms for proper prediction and isolation in different attack situations. As a result, the peak load point shifts after the attack, and also between the used machine learning algorithms, the two-layer neural network has significant accuracy for the presented model. The results of this research show that by using this machine learning algorithm, the system's security level has improved in the face of these damages.

A person with one eye missing, through various reasons, may suffer psychologically as well as physically. The loss of an eye can be solved cosmetically by an ocular implant. This artificial eye appears natural, but it is static. To let the artificial eye have the same natural movement as the real eye, an ocular system is developed. The artificial eye is mounted onto a tiny servomotor. The whole system shall be able to sense the real eye movement and control the motor to drive the artificial eye to the desired position. A tiny infrared sensor array is used for this study. This paper describes an approach of using the artificial neural network to do the sensor fusion to detect the eye movement. Two types of neural networks are used for the sensor fusion and sensor fault detection and recovery respectively. Usually the sensor fusion relies on the model of the system, however, sometimes it is not possible to get an accurate model of the system, or one or several of the parameters of the system may be unknown or partially known. In addition, there may be measurement inaccuracies associated with the sensors. In this case, the conventional method may not have a good performance. An artificial neural network can learn the characteristic of a non-linear, non-modeled system through training samples. Then during the real application, the sensor signal can be used to feed the network and obtain the desired output. Using the micro sensor array to detect the eye movement we carried out an experimental study. The sensor data is amplified and digitized then sent to the computer. Two-layer neural networks are trained by the data samples. First, the trained network is used for sensor fusion, and then two neural networks are used to detect the sensor failure and recover the faulty data respectively. Experimental studies with soft sensor failure and hard sensor failure are included. The main part of this paper deals with the network training method and further considerations.

This research focuses on optimizing a neural network model for stock price prediction using Particle Swarm Optimization (PSO), considering the inherent risks and potential high returns associated with stock investment. Given the challenges posed by stock price volatility, this study combines Moving Average (MA) a fundamental statistical technique in stock market analysis with advanced data mining approaches, specifically neural networks and PSO, to enhance prediction accuracy. The primary objective is to improve the efficiency of neural networks by minimizing error rates and equipping investors with more reliable tools for financial decision-making. The proposed methodology involves converting historical stock price data into a Simple Moving Average (SMA) over a 5-day period, followed by optimizing a neural network model using PSO. This optimization process fine-tunes key parameters, particularly the weight distributions of various stock market indicators, including Open SMA, High SMA, Low SMA, and Close SMA. Model performance is evaluated using Root Mean Square Error (RMSE) as a validation metric. The findings indicate a significant enhancement in the predictive accuracy of the neural network model after PSO optimization. The optimal configuration is identified in a two-layer neural network with a specific node arrangement. This optimized model not only improves stock price forecasting precision but also has practical implications for investors and financial analysts in risk management and profit maximization.

This paper presents a novel Stator Current based Model Reference Adaptive System (SC_MRAS) speed observer for high-performance Six Phases Induction Motor (SPIM) drives using linear neural network. The article aim is intended to improve performance of an SC_MRAS observer, which were presented in the literature. In this proposed scheme, the measured stator current components are used as the reference model of the MRAS observer to avoid the use of a pure integrator and reduce the influence of motor parameter variation. The adaptive model uses a two-layer Neural Network (NN) to estimate the stator current, which has been trained online by means of a Least Squares (LS) algorithm instead of uses a nonlinear Back Propagation Network (BPN) algorithm to reduce the complexity and computational burden, it also help to improve some disadvantages cause by the inherent nonlinearity of the BPN algorithm as local minima, two heuristically chosen parameters, initialization, and convergence problems, paralysis of the neural network. The adaptive model of the proposed scheme is employed in prediction mode, not in simulation mode as is usually the case in the literature, this made the proposed observer operate better accuracy and stability. In the proposed observer, stator and rotor resistance values are estimated online, these values thereafter were updated for the current observer and rotor flux identifier to enhance the accuracy, robustness and insensitivity to parameters variation for the proposed observer. The proposed LS SC_MRAS observer has been verified thought the simulation and compared with the BPN MRAS observer. The simulation results have proven that the speed is estimated a consequent quicker convergence, do not need the estimated speed filter, lower estimation errors both in transient and steady state operation, better behavior in low and zero speed operation.