Chapter 6 - Neural networks

The author proposes the scale-space transformation (SST) as a paradigm for information processing in biological neural networks. The SST concept includes scale-space, scale-time, and scale-space-time mappings. Hierarchical nonlinear (HNL) systems theory, together with the SST paradigm, causality requirements in the time domain, and uncertainty constraints in time and space domains, can be used to develop morphogenic models of biological neural networks. Since morphogenic models need only capture the functional modality of their physical counterparts, there may or may not be an observable resemblance to physical structure. To illustrate these concepts, the author discusses a morphogenic model of the mammalian visual system (MVS) in terms of SST mappings. As an example he uses an exponential retinotopic mapping, which is called the log Z SST (LZ SST). Using HNL and SST concepts, the author suggests a layered model of the MVS neural network.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Coarse-graining microscopic models of biological neural networks to obtain mesoscopic models of neural activities is an essential step towards multi-scale models of the brain. Here, we extend a recent theory for mesoscopic population dynamics with static synapses to the case of dynamic synapses exhibiting short-term plasticity (STP). The extended theory offers an approximate mean-field dynamics for the synaptic input currents arising from populations of spiking neurons and synapses undergoing Tsodyks–Markram STP. The approximate mean-field dynamics accounts for both finite number of synapses and correlation between the two synaptic variables of the model (utilization and available resources) and its numerical implementation is simple. Comparisons with Monte Carlo simulations of the microscopic model show that in both feedforward and recurrent networks, the mesoscopic mean-field model accurately reproduces the first- and second-order statistics of the total synaptic input into a postsynaptic neuron and accounts for stochastic switches between Up and Down states and for population spikes. The extended mesoscopic population theory of spiking neural networks with STP may be useful for a systematic reduction of detailed biophysical models of cortical microcircuits to numerically efficient and mathematically tractable mean-field models.

Network neuroscience is the emerging discipline concerned with investigating the complex patterns of interconnections found in neural systems, and identifying principles with which to understand them. Within this discipline, one particularly powerful approach is network generative modelling, in which wiring rules are algorithmically implemented to produce synthetic network architectures with the same properties as observed in empirical network data. Successful models can highlight the principles by which a network is organized and potentially uncover the mechanisms by which it grows and develops. Here, we review the prospects and promise of generative models for network neuroscience. We begin with a primer on network generative models, with a discussion of compressibility and predictability, and utility in intuiting mechanisms, followed by a short history on their use in network science, broadly. We then discuss generative models in practice and application, paying particular attention to the critical need for cross-validation. Next, we review generative models of biological neural networks, both at the cellular and large-scale level, and across a variety of species including Caenorhabditis elegans, Drosophila, mouse, rat, cat, macaque and human. We offer a careful treatment of a few relevant distinctions, including differences between generative models and null models, sufficiency and redundancy, inferring and claiming mechanism, and functional and structural connectivity. We close with a discussion of future directions, outlining exciting frontiers both in empirical data collection efforts as well as in method and theory development that, together, further the utility of the generative network modelling approach for network neuroscience.

This paper proposes a step-by-step technique for combining basic models that forecast electricity consumption in an artificial neural network by the method of preliminary selection and further hybridization. The reported experiments were conducted using data on hourly electricity consumption at the metallurgical plant AO ArcelorMittal Temirtau in the period from January 1, 2019, to November 30, 2021. The current research is related to the planned introduction of a balancing electricity market. 96 combinations of basic models were compiled, differing in the type of neural network, the set of initial data, the order of lag, the learning algorithm, and the number of neurons in the hidden layer. It has been determined that the NARX-type network is the most optimal architecture to forecast electricity consumption. Based on experimental studies, the number of hidden neurons needed to form a planned daily profile should equal 3 or 4; it is recommended to use the conjugate gradient method as a learning algorithm. When selecting models from three groups, it was revealed that the conjugate gradient method produces better results compared to the Levenberg-Marquardt algorithm. It is determined that the values of the selected RMSE error indicator take values of 23.17, 22.54, and 22.56, respectively, for the first, second, and third data groups. The adaptive hybridization method has been shown to reduce the RMSE error rate to 21.73. However, the weights of the best models with values of 0.327 for the first group of data, and 0.336 for the second and third ones, show that the individual use of a separate combination of models is also applicable. The devised forecasting electricity consumption model can be integrated into an automated electricity metering system

Neural networks have been studied for many years with the hope of achieving human-like performance in such fields as speech and image recognition. A recent resurgence has resulted from VLSI advances, neural network models, and learning algorithms. Neural networks are also very suitable in certain areas such as NP-complete constraint and satisfaction problems, due to the nature of parallel and distributed processing. Neural network models are composed of a mass of fairly simple computational elements and rich interconnections between the elements. Neural networks operate in a parallel and distributed fashion which may resemble biological neural networks. Behaviors of neurons and the strengths of synaptic interconnections are simulated by operational amplifiers and resistors respectively. Several examples are presented, including how to build neural network components based on analog circuits for simulating neural networks and conventional logic circuits.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Artificial neural network (ANN) models have been used in the past to model surface roughness in manufacturing processes. Specifically, different parameters influence surface roughness in fused filament fabrication (FFF) processes. In addition, the characteristics of the networks have a direct impact on the performance of the models. In this work, a study about the use of ANN to model surface roughness in FFF processes is presented. The main objective of the paper is discovering how key ANN parameters (specifically, the number of neurons, the training algorithm, and the percentage of training and validation datasets) affect the accuracy of surface roughness predictions. To address this question, 125 3D printing experiments were conducted changing orientation angle, layer height and printing temperature, and measuring average roughness Ra as response. A multilayer perceptron neural network model with backpropagation algorithm was used. The study evaluates the effect of three ANN parameters: (1) number of neurons in the hidden layer (4, 5, 6 or 7), (2) training algorithm (Levenberg–Marquardt, Resilient Backpropagation or Scaled Conjugate Gradient), and (3) data splitting ratios (70%–15%–15% vs. 55%–15%–30%). Mean Absolute Error (MAE) was used as the performance metric. The Resilient Backpropagation algorithm, 7 neurons, and using 55% of training data yielded the best predictive performance, minimizing the MAE. Additionally, the impact of the dataset size on prediction accuracy was analysed. It was observed that the performance of the ANN gets worse as the number of datasets is reduced, emphasizing the importance of having sufficient data. This study will help to select appropriate values for the printing parameters in FFF processes, as well as to define the characteristics of the ANN to be used to model surface roughness.

The purpose of the studyis to examine the theory and practice ofusing neural networks in education, to develop the concept of aneural network adaptive learning environment, and to implementa neural network model of one of the subsystems of this environment (using a model example of creating an adaptive educationaltrajectory).Materials and methods.The study includes a review of bibliographicsources on the application of neural networks in the field of education.It also includes modeling the structure of a neural network adaptivelearning environment. The PyTorch library was used to programmatically implement the neural network.Results.The prospects for the use of neural networks in the field ofeducation are considered, including various tasks of recognition,diagnostics, classification, clustering, forecasting, optimization,etc. A structural model of an adaptive learning NeuroSmartenvironment is created. This environment includes a number ofsubsystems for solving the following tasks: biometric identificationof the student; determining starting level; choosing an adaptivelearning path; determining the current level of competency development; automated verification of students’ work using recognitiontechnology; analysis of the final result of training; monitoringinformation security incidents in an electronic course, etc. Inorder to study the possibilities and problems of applying neuralnetwork models to the tasks of constructing student adaptivelearning trajectories, a model example of a neural network wascreated. This example illustrates the possibility of using a neural network to select further nodes of the educational trajectorybased on the available data on the current learning parametersin an electronic course. To implement the neural network, thePyTorch deep learning library and Pandas library modules wereused. SGD, Adam, Rprop were used as an optimizer to performgradient descent steps. For each of the optimizers, a computerstudy of the stability of the neural network was carried out byvarying the following parameters: the learning rate coefficient,the number of neurons in the hidden layer, and the number oftraining epochs.Conclusion.It can be assumed that the next stage in the evolutionof the use of neural network technologies in the field of educationwill be their integration into complex multi-component Smart systemscapable of automatically supporting student learning at all stagesof the implementation of their personal educational trajectory. Obviously, the practical implementation of complex neural networksystems of this level is a very difficult task and can be solved so faronly at the level of individual subsystems. There are a number of problems associated with computer simulation of the educationalenvironment based on neural network models: the question of theoptimal structure of the neural network (for example, the numberof neurons and layers in the network) has not been studied enough,there are no clear criteria for the optimality of the adaptive educational trajectory. Nevertheless, it should be noted that the task ofdeveloping new forms and technologies of personalized e-learning isin great demand, which makes modeling based on neural networksespecially relevant.

&amp;lt;p&amp;gt;Measuring and assessing the bedload data is a crucial for successful and efficient river management. Hence, the information about the bedload transport and characteristics helps to describe the dynamics of the river morphology and to evaluate the impacts on boat navigation, hydropower production, ecological systems and aquatic habitat.&amp;lt;/p&amp;gt;&amp;lt;p&amp;gt;Although the acoustic Doppler current profilers are designed to measure water velocities and discharges, they have been successfully used to measure some bedload characteristics, such as the apparent bedload velocity. The correlation between the apparent bedload velocity and the bedload transport rates measured by physical bedload samplers (e.g. pressure difference) has been examined and relatively high correlations have been reported. Moreover, laboratory experiments have proven that there is a strong correlation between the bedload concentration and particle size distribution and corrected backscattering strength obtained from the ADCPs.&amp;lt;/p&amp;gt;&amp;lt;p&amp;gt;The bedload transport rates yielded from the ADCPs outputs are usually derived as regression model-fitting of the measured apparent velocity and the physically collected bedload samples at the same time and position.&amp;amp;#160; Alternatively, a semi-empirical kinematical approach is used, where the apparent bedload velocity is the main component and the bedload concentration is empirically estimated. However, the heterogeneous and sporadic motion of the bedload particles is often followed by high uncertainty and weak performance of these approaches.&amp;lt;/p&amp;gt;&amp;lt;p&amp;gt;Machine learning offers a relatively simple and robust method that has the potential to describe the nonlinearity of the complex bedload motion and so far, it has not been previously exploited for predicting transport rates. This study implements artificial neural network techniques to develop a model for predicting bedload transport rates by using only ADCP data outputs as training data. Data processing techniques are used to extract relevant features from the corrected backscattering strength and apparent velocity obtained from the ADCPs. More than 60 features were derived in the ADCPs dataset, and the most relevant features are selected through neighborhood component analysis. These features are used as inputs in conventional supervised neural network architecture which consists of two hidden layers and 35 neurons. This model is used to capture the distribution of the ADCP features for each output (e.g., physically measured transport rates and grain size from bedload samples) in the training sample. The back-propagation algorithm (BPA) is still one of the most widely used learning algorithms in the training process and thus herein applied. The learning rate, number of neurons and hidden layers were optimized by using Bayesian optimization techniques. The network was trained with more than 60 bedload samples and corresponding 5 - 10 min time series of ADCP preprocessed data. The rest of the samples were used for validation of the model. The validation resulted in correlation coefficients higher than 0.9 and the, which is significantly higher value than the corresponding values for the methodologies developed before. Aiming to develop a more robust and stable ANN model, further testing of different training algorithms must be performed, different ANN architecture should be tested, and more data shall be included.&amp;lt;/p&amp;gt;

Backpropagation is the most popular algorithm for training neural networks. However, this gradient-based training method is known to have a tendency towards very long training times and convergence to local optima. Various methods have been proposed to alleviate these issues including, but not limited to, different training algorithms, automatic architecture design and different transfer functions. In this chapter we continue the exploration into improving gradient-based learning algorithms through dynamic transfer function modification. We propose opposite transfer functions as a means to improve the numerical conditioning of neural networks and extrapolate two backpropagation-based learning algorithms. Our experimental results show an improvement in accuracy and generalization ability on common benchmark functions. The experiments involve examining the sensitivity of the approach to learning parameters, type of transfer function and number of neurons in the network.

The objective of this study is to show how a multi-layer perceptron (MLP) neural network can be used to model a CMM measuring process. To date, most MLP-based process models have been established for process mean only. An innovative approach is proposed to model simultaneously the mean and the variation of a CMM process using one integrated MLP architecture. Therefore, the MLP-based model obtained captures not only the process mean but also the process variation information. Selected issues related to neural network training are also discussed. Specifically, the guideline that was proposed by Mirchandani and Cao (1989) for selecting a number of hidden neurons is tested to determine the effects of the number of hidden neurons. The performances of two different learning algorithms - back-propagation with momentum factor (BPM) and the Fletcher-Reeves (FR) algorithm - are studied in terms of CPU time, training error, and generalization error.

Besides the feedforward neural networks, there are the recurrent networks, where the impulses can be transmitted in both directions due to some reaction connections in these networks. Recurrent Neural Networks (RNNs) are linear or nonlinear dynamic systems. The dynamic behavior presented by the recurrent neural networks can be described both in continuous time, by differential equations and at discrete times by the recurrence relations (difference equations). The distinction between recurrent (or dynamic) neural networks and static neural networks is due to recurrent connections both between the layers of neurons of these networks and within the same layer, too. The aim of this chapter is to describe a Recurrent Fuzzy Neural Network (RFNN) model, whose learning algorithm is based on the Improved Particle Swarm Optimization (IPSO) method.

This paper presents a versatile hyper-ellipsoidal basis function for function approximation in a given high dimensional space. This hyper-ellipsoidal basis function can be translated and rotated to cover the data based upon the distribution of data in a given high dimensional space. Based on this function, we propose a one-pass hyper-ellipsoidal learning algorithm for which any new incoming data can be fed for learning without involving the previously learned one. This learning algorithm is used to adjust the parameters of the versatile hyper-ellipsoidal basis function. In addition, we propose the hyper-ellipsoidal basis function (HEBF) neural network that uses the one-pass hyper-ellipsoidal neural learning algorithm. The structure of this neural network is similar to the radial basis function (RBF) neural networks. The hidden neurons in the HEBF neural network can be increased or decreased during learning process. The number of the hidden neurons in the network can be grown based on geometric growth criterion and can be reduced by merging the two hidden neurons into a new hidden neuron based on merging criterion during learning process. The merging process can be done independently without considering the learned data set.KeywordsRadial Basis FunctionHide NeuronRadial Basis Function Neural NetworkHigh Dimensional SpaceInitial WidthThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

More accurate load forecasting is more and more important for economic generation and system security in power systems. In this paper, an ant colony algorithm based feed forward neural network model to integrate maximum correlation theory and deterministic annealing clustering technique for load forecasting problem is proposed. As we all know, if so many load influential factors are consider as network input neurons, it will result in complicated structure of network, long learning time and inaccurate prediction. So, first, maximum correlation method is used for selecting main load influential factors which have more relevant to load parameter. It can reduce irrelevant load influential factors and the input variables of the input layer for neural network. Next, deterministic annealing (DA) clustering technique is applied for dividing the practical load data into representative groups in order to reduce training time. Finally, the feed forward neural network with ant colony training algorithm is provided for short term load forecasting problem since this training algorithm can effective avoid local optimum and improve training accuracy. The experiments and the performance with presented neural network model are given, and the results showed the method could provide a satisfactory improvement of the forecasting accuracy.

A connectionist fuzzy classifier, called CFC, was proposed and shown to perform well in speech recognition. In this article a new learning algorithm with better on-line learning ability and speech recognition performance is developed to replace the original learning algorithm of the CFC model. To distinguish the two learning algorithms, the CFC executing the new learning algorithm is called modified CFC MCFC. Both of the learning algorithms execute a one-pass learning scheme, which is far faster than the backpropagation-based learning algorithms. In addition, both the CFC and MCFC employ the same four-layered feed forward network structure to implement a “weighted Euclidean distance” fuzzy classification procedure that can be realized by digital hardware with a massively parallel architecture. Some experiments and comparisons for MCFC, CFC, and some other neural network models are also made. The experimental results show that the MCFC has better accuracy and stability for speech recognition than the original CFC, probabilistic neural network, fuzzy ART, and fuzzy ARTMAP, especially in a noisy environment.

The purpose of this study is to familiarise the reader with the foundations of neural networks. Artificial Neural Networks (ANNs) are algorithm-based systems that are modelled after Biological Neural Networks (BNNs). Neural networks are an effort to use the human brain's information processing skills to address challenging real-world AI issues. The evolution of neural networks and their significance are briefly explored. ANNs and BNNs are contrasted, and their qualities, benefits, and disadvantages are discussed. The drawbacks of the perceptron model and their improvement by the sigmoid neuron and ReLU neuron are briefly discussed. In addition, we give a bird's-eye view of the different Neural Network models. We study neural networks (NNs) and highlight the different learning approaches and algorithms used in Machine Learning and Deep Learning. We also discuss different types of NNs and their applications. A brief introduction to Neuro-Fuzzy and its applications with a comprehensive review of NN technological advances is provided.