Neural network architectures and learning

Abstract-Hebbian learning is widely accepted in the fields of psychology, neurology, and neurobiology. It is one of the fundamental premises of neuroscience. The LMS (least mean square) algorithm of Widrow and Hoff is the world's most widely used adaptive algorithm, fundamental in the fields of signal processing, control systems, pattern recognition, and artificial neural networks. These are very different learning paradigms. Hebbian learning is unsupervised. LMS learning is supervised. However, a form of LMS can be constructed to perform unsupervised learning and, as such, LMS can be used in a natural way to implement Hebbian learning. Combining the two paradigms creates a new unsupervised learning algorithm that has practical engineering applications and provides insight into learning in living neural networks. A fundamental question is, how does learning take place in living neural networks? Nature's little secret, the learning algorithm practiced by nature at the neuron and synapse level, may well be the Hebbian-LMS algorithm.

Neural networks provide a solution to overcome drawbacks of the quantitative fault diagnosis because first, they are capable to model off-line the behaviour of linear and non-linear systems. Secondly, they can also learn on-line the behaviour of a system requiring no priori knowledge about the system. The neural networks are particularly good for fault diagnosis of systems that have imperfect models and/or incomplete data. There are two basic learning methods of neural networks that are applied to fault diagnosis: supervised and unsupervised learning methods. To solve problem of priori unknown faults, unsupervised learning is used. The Counterpropagation network was selected to diagnose faults as result of analysis of supervised and unsupervised learning methods applied to neural networks.

The investigation of nonlinear processes in brain systems based on the oscillatory-wave approach is currently one of the most consequential domains of researching signal generation and processing mechanisms in the brain. These investigations may offer insights into various phenomena, such as the mechanism of associative memory, based on network models of biologically relevant neurons. A neural network architecture is proposed in this paper to solve applied problems, such as signal filtering, processing, and recognition of information images. Mechanisms for incomplete fragments were implemented, enabling the extraction and complete restoration of objects from memory. Each neuron in this neural network comprises a Hodgkin–Huxley biophysical model with a Mainen modification, the dynamics of which most plausibly reproduce the processes in the neural cells of the brain. In the research on individual neurons, a constant external current with varying amplitudes was applied. The study determined the average periods of self-oscillations and identified parameter values for the Andronov–Hopf bifurcation. In particular, a stable limit cycle is created and destroyed depending on different scenarios when the bias current is increased or decreased. A stable limit cycle arises via the Andronov–Hopf bifurcation and is terminated via the saddle-node bifurcation on the cycle. The study identified parameter regions whereby two neurons with excitatory and inhibitory synaptic connections synchronize. The neurons were all in self-oscillating mode and exhibited a stable limit cycle in the phase space. A three-layer neural network was constructed, comprising of a reference neuron, sensory, control, and interneurons. The control and intermediate layer connections are arranged based on the Hebbian learning rule. Areas of excitation for various types of neurons were found. Experiments were conducted using the neural network architecture to distinguish binary information patterns encoded through signal phase. Excitatory and inhibitory synaptic connections were employed to encode the images stored in memory. The dynamic patterns are determined through the in-phase or anti-phase mode of phase locking, i.e., phase synchronization relative to the global rhythm. Before reaching the output layer of neurons, the signals are summed on the interneuron layer with the signal of the reference neuron, resulting in the filtering out of a certain information segment (in-phase or anti-phase). The network accurately recognized the patterns after parameter adjustments were made. Areas of neuron phase synchronization were discovered, enabling control of network activity modes. The confirmation of the viability of phase capture and pattern recognition modes using Hodgkin–Huxley–Mainen neurons was achieved. In total, as a result of the implementation of the spike neural network modeling project: 1) two-parameter diagrams of regions of neuronal excitation in various modes are calculated; 2) the effect of excitatory and inhibitory coupling in synaptic currents is used to represent the input signal; 3) an abstract mathematical algorithm for calculating the Hebbian matrix has been transferred to the implementation of synaptic currents between layers of neurons; 4) the effect of phase clusters is used to represent the output pattern; 5) the neurons’ topology in the visual-cerebral department served as a working model for recognizing graphic images through the Hopfield network on a spike neural network, using the Hebbian rule.

Hybrid feedback Group Method of Data Handling (GMDH)-type neural network self-selecting various neurons is applied to the computer aided image diagnosis (CAD) of lung cancer. In this algorithm, three types of neural networks, such as sigmoid function neural network, radial basis function (RBF) neural network and polynomial neural network, can be generated using three types of neuron architectures, and the neural network architecture which is the most fit the complexity of medical images, is selected from these three neural network architectures. Furthermore, this GMDH-type neural network has feedback loop and the structural parameters such as the number of feedback loops, the number of neurons in hidden layers and the relevant input variables are automatically selected so as to minimize the prediction error criterion. This GMDH-type neural network is applied to CAD of lung cancer and two kinds of GMDH-type neural networks which recognize lung regions and lung cancer regions, are automatically generated from the multi-detector row CT (MDCT) images of lungs. It is shown that this GMDH-type neural network is useful for CAD of lung cancer.

Utilizing the enormous potential of quantum computers requires new and practical quantum algorithms. Motivated by the success of machine learning, we investigate the fusion of neural and quantum computing, and propose a learning method for a quantum neural network inspired by the Hebb rule. Based on an analogy between neuron-neuron interactions and qubit-qubit interactions, the proposed quantum learning rule successfully changes the coupling strengths between qubits according to training data. To evaluate the effectiveness and practical use of the method, we apply it to the memorization process of a neuro-inspired quantum associative memory model. Our numerical simulation results indicate that the proposed quantum versions of the Hebb and anti-Hebb rules improve the learning performance. Furthermore, we confirm that the probability of retrieving a target pattern from multiple learned patterns is sufficiently high.

This paper proposes a position control strategy based on artificial neural networks (ANN) in the face of structured and unstructured dynamic uncertainties. The control structure consists of a feedforward multilayer perceptron (MLP) to approximate the manipulator's inverse dynamics online, a feedback radial basis function (RBF) neural network to compensate for the residual errors, and a reference model that defines the desired error dynamics. The online adaptation of the RBF neural network is is accomplished through two methods: (i) the least mean squares (LMS), and (ii) the recursive least squares (RLS) algorithms. A comparison study is conducted to evaluate the efficiency of both algorithms on the tracking ability of the proposed control scheme. Simulation results highlight the performance of the proposed control structures in compensating for the highly nonlinear unknown dynamics of the manipulator and its robustness in the presence of model imperfections.

In this paper we introduce a novel method for solving the trajectory tracking problem for a quadrotor system based on backstepping control and radial basis function (RBF) neural networks. Backstepping controllers in quadrotor control are designed so as to guarantee Lyapunov stability of the closed-loop system based on a dynamic model of the quadrotor, derived using first principles equations. Such a model is prone to modeling errors due to various factors, including uncertainties, time varying quadrotor parameters and external disturbances acting on the vehicle, thus compromising the controller performance. To remedy this situation, we couple the controller with a radial basis function (RBF) neural network trained with the fuzzy means (FM) algorithm, which allows the proposed RBF – backstepping framework to take into account unmodeled dynamics with increased accuracy, thus improving the tracking performance. The resulting method is evaluated on a detailed quadrotor model over trajectories with distinct geometrical characteristics, and compared with a backstepping controller using a different neural network architecture and a standard backstepping control scheme. The proposed control method produces very competitive results, outperforming its rival in terms of trajectory tracking performance.

An algorithm called Incremental Best Estimate Directed Search (IBEDS) was proposed for Takagi-Sugeno type fuzzy logic controller (FLC) automatic optimization. IBEDS starts with an initial training set that may be empty, an FLC with randomly initialized rule output parameters is trained by least mean square (LMS) learning algorithm in an iterative procedure. In each iteration, the trained FLC is evaluated with cell state space based global and local performance measures, the training set is then updated based on the evaluation under best kept policy, which only keeps the best control commands found so far. In this way, the training set is optimized in every iteration, and the FLC trained by the training set is also optimized progressively. The philosophy behind IBEDS is that the parameters of an FLC do not represent the performance of the controller directly although the goal of the optimization is to find an optimal controller parameter set so that the controller can have optimal global performance. However, the parameter set of an FLC with suboptimal performance may well represent a point in the parameter search space where the optimal one is nearby. Therefore the search algorithm should put more efforts in that area. LMS learning error is discarded in IBEDS as it does not represent the performance of the trained FLC either. This learning error can be reused when the training set is highly optimized and an exact approximation of the training set by an FLC is desired. To further speed up the optimization, in addition to suggesting the reuse of the LMS learning error, this paper also proposes to reuse the FLC parameter set during a search. A 4D inverted pendulum is studied, and the simulation results show that the search speed is improved.

University admissions and business personnel offices use a limited number of resources to process an ever-increasing quantity of student and employment applications. Application systems are further constrained to identify and acquire, in a limited time period, those candidates who are most likely to accept an offer of enrolment or employment. Neural networks are a new methodology to this particular domain. Various neural network architectures and learning algorithms are analyzed comparatively to determine the applicability of supervised learning neural networks to the domain problem of personnel resource allocation and to identify optimal learning strategies in this domain. This paper focuses on multilayer perceptron backpropagation, radial basis function, counterpropagation, general regression, fuzzy ARTMAP, and linear vector quantization neural networks. Each neural network predicts the probability of enrolment and nonenrolment for individual student applicants. Backpropagation networks produced the best overall performance. Network performance results are measured by the reduction in counsellors student case load and corresponding increases in student enrolment. The backpropagation neural networks achieve a 56% reduction in counsellor case load.

Machine learning, and in particular neural networks, require large amounts of data, which is increasingly highly distributed (e.g. over user devices, or independent storage systems). Aggregating this data at one site for learning can be unpractical due to network costs, legal constraints, or privacy concerns. Decentralized machine learning holds the potential to address these concerns, but unfortunately, most of the approaches proposed so far for distributed learning with neural networks are mono-task, and do not transfer easily to multi-task problems. In this paper, we propose a novel learning method for neural networks that is decentralized, multi-task, and that keeps users’ data local. Our approach works with different learning algorithms, on various types of neural networks. We formally analyze the convergence of our method, and we evaluate its efficiency in a range of neural networks and learning algorithms, demonstrating its benefits in terms of learning quality and convergence.

We had previously shown that regularization principles lead to approximation schemes that are equivalent to networks with one layer of hidden units, called regularization networks. In particular, standard smoothness functionals lead to a subclass of regularization networks, the well known radial basis functions approximation schemes. This paper shows that regularization networks encompass a much broader range of approximation schemes, including many of the popular general additive models and some of the neural networks. In particular, we introduce new classes of smoothness functionals that lead to different classes of basis functions. Additive splines as well as some tensor product splines can be obtained from appropriate classes of smoothness functionals. Furthermore, the same generalization that extends radial basis functions (RBF) to hyper basis functions (HBF) also leads from additive models to ridge approximation models, containing as special cases Breiman's hinge functions, some forms of projection pursuit regression, and several types of neural networks. We propose to use the term generalized regularization networks for this broad class of approximation schemes that follow from an extension of regularization. In the probabilistic interpretation of regularization, the different classes of basis functions correspond to different classes of prior probabilities on the approximating function spaces, and therefore to different types of smoothness assumptions. In summary, different multilayer networks with one hidden layer, which we collectively call generalized regularization networks, correspond to different classes of priors and associated smoothness functionals in a classical regularization principle. Three broad classes are (1) radial basis functions that can be generalized to hyper basis functions, (2) some tensor product splines, and (3) additive splines that can be generalized to schemes of the type of ridge approximation, hinge functions, and several perceptron-like neural networks with one hidden layer.

Introduction. The article considers the solution of the problem of detecting fires at technological facilities in automatic mode. To solve this problem, an approach was chosen to create a convolutional neural network capable of operating on a real-time video stream. Aims and Purposes. The aim of this work is to create a neural network capable of detecting flames and smoke in the image from CCTV cameras. The purposes of the work include: selection of the optimal architecture of the neural network in accordance with the latest research in this field; speeding up the chosen architecture using quantization and filter thinning techniques. Methods. Different architectures of convolutional neural networks performing the task of detecting objects in an image are considered. Their performance and quality of work are compared. The YOLOv5 architecture, its target function, training methods and ways of speeding up work are considered. Results and discu ssion. The paper shows the results of training a convolutional neural network of the YOLOv5 architecture for the task of flame and smoke detection, as well as how the results change when applying neural network acceleration methods. It was determined that the use of such acceleration methods such as quantization and filter cleaning can significantly increase up the speed of the neural network, while almost no loss in accuracy of operation. Conclusions. As a result of the conducted work, the architecture of the neural network was determined for performing the task of detecting a fire source. Based on the chosen architecture, a neural network was trained to detect flames and smoke in the image. The speed of its work allows to process video stream in real-time without using graphic accelerator.

An adaptive array code acquisition for direct-sequence/ code-division multiple access (DS/CDMA) systems was recently proposed to enhance the performance of the conventional correlator-based method. The scheme consists of an adaptive spatial and an adaptive temporal filter, and can simultaneously perform beamforming and code-delay estimation. Unfortunately, the scheme uses a least-mean-square (LMS) adaptive algorithm, and its convergence is slow. Although the recursive-least-squares (RLS) algorithm can be applied, the computational complexity will greatly increase. In this paper, we solve the dilemma with a low-complexity conjugate gradient (LCG) algorithm, which can be considered as a special case of a modified conjugate gradient (MCG) algorithm. Unlike the original conjugate gradient (CG) algorithm developed for adaptive applications, the proposed method, exploiting the special structure inherent in the input correlation matrix, requires a low computational-complexity. It can be shown that the computational complexity of the proposed method is on the same order of the LMS algorithm. However, the convergence rate is improved significantly. Simulation results show that the performance of adaptive array code acquisition with the proposed CG algorithm is comparable to that with the original CG algorithm.

Although numerical calculations of heat transfer and fluid flow can provide detailed insights into welding processes and welded materials, these calculations are complex and unsuitable in situations where rapid calculations are needed. A recourse is to train and validate a neural network, using results from a well tested heat and fluid flow model to significantly expedite calculations and ensure that the computed results conform to the basic laws of conservation of mass, momentum and energy. Seven feedforward neural networks were developed for gas metal arc (GMA) fillet welding, one each for predicting penetration, leg length, throat, weld pool length, cooling time between 800°C and 500°C, maximum velocity and peak temperature in the weld pool. Each model considered 22 inputs that included all the welding variables, such as current, voltage, welding speed, wire radius, wire feed rate, arc efficiency, arc radius, power distribution, and material properties such as thermal conductivity, specific heat and temperature coefficient of surface tension. The weights in the neural network models were calculated using the conjugate gradient (CG) method and by a hybrid optimisation scheme involving the CG method and a genetic algorithm (GA). The neural network produced by the hybrid optimisation model produced better results than the networks based on the CG method with various sets of randomised initial weights. The CG method alone was unable to find the best optimal weights for achieving low errors. The hybrid optimisation scheme helped in finding optimal weights through a global search, as evidenced by good agreement between all the outputs from the neural networks and the corresponding results from the heat and fluid flow model.

Inhaling particulate matter such as PM2.5 can have a hazardous impact on the human health. In order to predict the PM2.5 concentration, Artificial Neural Networks trained with conjugate gradient descent such as Multi Layer Perceptron (MLP), cascade forward neural network, Elman neural network, Radial Basis Function (RBF) neural network and Non-linear Autoregressive model with exogenous input (NARX) along with regression models such as Multiple Linear Regression (MLR) consisting of batch gradient descent, stochastic gradient descent, mini-batch gradient descent and conjugate gradient descent algorithms and Support Vector Regression (SVR) were implemented. In these models, the concentration of PM2.5 was the dependent variable and the data related to concentrations of PM2.5, SO2, O3 and meteorological data including average Maximum Temperature (MAX T), daily wind speed (WS) for the years 2010–2016 in Houston and New York were the independent variables. For the final forecast, several ensemble models of individual neural network predictors and individual regression predictors have been presented.