On the Circuit Complexity of Sigmoid Feedforward Neural Networks

This paper examines the circuit complexity of feedforward neural networks having sigmoid activation function. The starting point is the complexity class NN k defined in [18]. First two additional complexity classes N N Δ k https://s3-euw1-ap-pe-df-pch-content-public-p.s3.eu-west-1.amazonaws.com/9781315784076/1ea703af-0c66-41e4-8ff3-9e9c700ac445/content/eq1772.tif/> and N N Δ , e k https://s3-euw1-ap-pe-df-pch-content-public-p.s3.eu-west-1.amazonaws.com/9781315784076/1ea703af-0c66-41e4-8ff3-9e9c700ac445/content/eq1773.tif/> having less restrictive conditions (than NN k ) concerning fan-in and accuracy are defined. We then prove several relations among these three classes and well established circuit complexity classes. All the proofs are constructive as being built around binary adders. By relaxing the fan-in condition from logarithmic [18] to (almost) polynomial, the results show that substantially larger classes of sigmoid activation feedforward neural networks can be implemented in polynomial size Boolean circuits. This is done on the expense of a logarithmic factor increase in the number of layers, but having constant fan-in (of 2). The main conclusion is that there are interesting fan-in dependent depth-size tradeoffs when trying to digitally implement sigmoid activation neural networks.

The global increase in greenhouse gas emissions from automobiles has brought about the manufacture and usage of large quantities of electric vehicles (EVs). However, to ensure proper integration of EVs into the grid, there is a need to forecast the charging demand of EVs accurately. This paper presents a short-term electric vehicle charging demand forecast using a feedforward artificial neural network optimized with a modified local leader phase spider monkey optimization (MLLP-SMO) algorithm, a proposed variant of spider monkey optimization. A proportionate fitness selection is employed to improve the update process of the local leader phase of the spider monkey optimization. The proposed algorithm trains a feedforward neural network to forecast electric vehicle charging demand. The effectiveness of the proposed forecasting model was tested and validated with electric vehicle public charging data from the United Kingdom Power Networks Low Carbon London Project. The model's performance was compared to a feedforward neural network trained with particle swarm optimization, genetic algorithm, classical spider monkey optimization, and two conventional forecasting models, multi-linear regression and Monte Carlo simulation. The performance of the proposed forecasting model was assessed using the mean absolute percentage error of forecast and forecasting accuracy. The model produced a forecast accuracy and mean absolute percentage error of 99.88% and 3.384%, respectively. The results show that MLLP-SMO as a trainer predicted better than the other forecasting models and met industry standard forecast accuracy.

Single-hidden-layer feed-forward quantum neural network based on Grover learning

The authors examine the usefulness of the feedforward neural network as a controller. In particular, the authors consider as an example the problem of design of a power system stabilizer that has the structure of a classical feedforward artificial neural network. In this case, the controller is developed for a class of problems associated with linear plants. Certain observations are made regarding the generic behavior of systems with this type of controller structure. The authors incorporate a robustness criterion into the design as well as a conventional dynamic performance criterion.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

&lt;p&gt;This paper presents an inverse dynamic model estimation based on an artificial neural network of a complete new parallel robot manipulator prototype 6- PGK with six degrees of freedom, built at Petru Maior University of Tirgu-Mures. The model estimation of the parallel robot manipulator is performed with a feedforward artificial neural network. In the control engineering domain there are control structures that need the direct or inverse model of the process for ensuring the process control at the imposed performances. Usually, the determination of the direct/inverse mathematical model is a difficult or impossible task to be achieved. In these cases different non-parametric or parametric, off-line or on-line identification methods are used. A solution that may support the on-line parametric methods is represented by the feedforward artificial neural networks. By implementing feedforward artificial neural networks as a nonlinear autoregressive model with exogenous inputs, the authors investigate the possibility of choosing the optimum parameters that characterize the neural network so that it approximates as better as possible the model of the 6-PGK prototype robot. Finally an innovative algorithm is developed for obtaining the optimal configuration parameters set of the feedforward artificial neural network. The proposed algorithm helps in setting the optimal parameters of the neural network that offer high opportunities to provide satisfactory identification of the robot model. Experimental results obtained by a structure derived from the proposed solution demonstrate a good approximation related to the studied system, which is characterized by nonlinearities and high complexity.&lt;/p&gt;

The increase of the world population, especially in the global competition, together with the increasing use of fossil fuel resources to meet energy needs, leads to more greenhouse gases (more than one CO2, methane etc.) emissions and the global climate crisis. In this process, changes in meteorological events such as temperature, precipitation, and wind, attract attention moreover but when considered as a whole, we know that these negative changes in the ecosystem negatively affect many living groups. Sea Surface Temperature (SST) as measured meteorologically is the most important environmental parameter where these changes are monitored and observed. It draws attention to the fact that changes in SST are not limited to living organisms as habitats, but also catalyze many chain reactions, especially socio-economic impacts. Therefore, much of the work is devoted to forecasting studies to adapt to changing habitats and take the necessary precautions against potential risks. Feed-forward artificial neural networks have been commonly used to address these emerging needs. Artificial neural networks, which are a simple imitation of the human neurological system, have been used as an artificial intelligence method in forecasting problems due to their superior performance and not having the limitations of classical time series. In this study, the forecasting of the time series of monthly mean SST temperature obtained from Rize station between the years 2010 and 2020 is performed by using feed-forward artificial neural networks, and the forecasting performance of the corresponding time series is compared with many forecasting methods with different characteristics. The comparison of the methods used the mean square error and mean absolute percentage error criteria, which are commonly used in the forecasting literature. The analysis results showed that the analysis results obtained with the feed-forward artificial neural networks have the best prediction performance. As a result, it can be stated that the sea surface temperature can be forecasted with a very high accuracy using the feed-forward artificial neural networks.

Training set parallelism and network based parallelism are two popular paradigms for parallelizing a feedforward (artificial) neural network. Training set parallelism is particularly suited to feedforward neural networks with backpropagation learning where the size of the training set is large in relation to the size of the network. This paper analyzes training set parallelism for feedforward neural networks when implemented on a transputer array configured in a pipelined ring topology. Theoretical expressions for the time per epoch (iteration) and optimal size of a processor network are derived when the training set is equally distributed among the processing nodes. These show that the speed up is a function of the number of patterns per processor, communication overhead per epoch and the total number of processors in the topology. Further analysis of how to optimally distribute the training set on a given processor network when the number of patterns in the training set is not an integer multiple of the number of processors, is also carried out. It is shown that optimal allocation of patterns in such cases is a mixed integer programming problem. Using this analysis it is found that equal distribution of training patterns among the processors is not the optimal way to allocate the patterns even when the training set is an integer multiple of the number of processors. Extension of the analysis to processor networks comprising processors of different speeds is also carried out. Experimental results from a T805 transputer array are presented to verify all the theoretical results.

BACKGROUNDTraditional methods of developing predictive models in inflammatory bowel diseases (IBD) rely on using statistical regression approaches to deriving clinical scores such as the Crohn's disease (CD) activity index. However, traditional approaches are unable to take advantage of more complex data structures such as repeated measurements. Deep learning methods have the potential ability to automatically find and learn complex, hidden relationships between predictive markers and outcomes, but their application to clinical prediction in CD and IBD has not been explored previously.AIMTo determine and compare the utility of deep learning with conventional algorithms in predicting response to anti-tumor necrosis factor (anti-TNF) therapy in CD.METHODSThis was a retrospective single-center cohort study of all CD patients who commenced anti-TNF therapy (either adalimumab or infliximab) from January 1, 2010 to December 31, 2015. Remission was defined as a C-reactive protein (CRP) < 5 mg/L at 12 mo after anti-TNF commencement. Three supervised learning algorithms were compared: (1) A conventional statistical learning algorithm using multivariable logistic regression on baseline data only; (2) A deep learning algorithm using a feed-forward artificial neural network on baseline data only; and (3) A deep learning algorithm using a recurrent neural network on repeated data. Predictive performance was assessed using area under the receiver operator characteristic curve (AUC) after 10× repeated 5-fold cross-validation.RESULTSA total of 146 patients were included (median age 36 years, 48% male). Concomitant therapy at anti-TNF commencement included thiopurines (68%), methotrexate (18%), corticosteroids (44%) and aminosalicylates (33%). After 12 mo, 64% had CRP < 5 mg/L. The conventional learning algorithm selected the following baseline variables for the predictive model: Complex disease behavior, albumin, monocytes, lymphocytes, mean corpuscular hemoglobin concentration and gamma-glutamyl transferase, and had a cross-validated AUC of 0.659, 95% confidence interval (CI): 0.562-0.756. A feed-forward artificial neural network using only baseline data demonstrated an AUC of 0.710 (95%CI: 0.622-0.799; P = 0.25 vs conventional). A recurrent neural network using repeated biomarker measurements demonstrated significantly higher AUC compared to the conventional algorithm (0.754, 95%CI: 0.674-0.834; P = 0.036).CONCLUSIONDeep learning methods are feasible and have the potential for stronger predictive performance compared to conventional model building methods when applied to predicting remission after anti-TNF therapy in CD.

Weight initialization of sigmoidal feed forward artificial neural network (SFFANN) and the Convolutional neural networks (CNN) has been a known factor which affects the learning abilities of the neural network. The uniform random weight initialization approach has been quite often used as the conventional network weight initial technique, due to its simplicity. However, various researches have shown that the random technique may not be the ideal choice of weight initialization for these neural networks. In this work, we analyze two separate chaotic functions and explore the possibilities of these being used as the weight initialization methods against the conventional random initialization technique for SFANNs as well as for the CNNs. For the SFFANNs, this analysis were done over 8 function approximation problems chosen for experimentation. The mean test error values along with a two sample t-test results strongly suggest that the Chebyshev chaotic map based weight initialization technique outperforms the conventional random initialization technique for most of the problems under consideration and hence may be used as an alternative weight initialization technique for the SFFANNs. For the CNN experiment, the MNIST dataset was used for analyzing the performance of the random and the Chebyshev based initialization scheme. Results strongly support the use of the Chebyshev chaotic map based initialization scheme as an alternate to the conventional random initialization.

Circuit Complexity and Feedforward Neural Networks: Ian Parberry

Neural networks usually work adequately on small problems but can run into trouble when they are scaled up to problems involving large amounts of input data. Circuit Complexity and Neural Networks addresses the important question of how well neural networks scale - that is, how fast the computation time and number of neurons grow as the problem size increases. It surveys recent research in circuit complexity (a robust branch of theoretical computer science) and applies this work to a theoretical understanding of the problem of scalability. Most research in neural networks focuses on learning, yet it is important to understand the physical limitations of the network before the resources needed to solve a certain problem can be calculated. One of the aims of this book is to compare the complexity of neural networks and the complexity of conventional computers, looking at the computational ability and resources (neurons and time) that are a necessary part of the foundations of neural network learning. Circuit Complexity and Neural Networks contains a significant amount of background material on conventional complexity theory that will enable neural network scientists to learn about how complexity theory applies to their discipline, and allow complexity theorists to see how their discipline applies to neural networks.

A multi-layer feed forward neural network model for accurate prediction of flue gas sulfuric acid dew points in process industries

Characteristic features of feedforward artificial neural networks, acting as universal function approximators, are presented. The problem under consideration concerns inverse kinematics of a two-link planar manipulator. As shown in the article, a two-layer, feedforward neural network is able to learn the nonlinear mapping between the end-effector position domain and the joint angle domain of the manipulator. However, the necessary condition for achieving the required approximation quality is the selection of suitable network structure, especially with regard to the number of nonlinear, sigmoidal units in its hidden layer. Effects of learning algorithm and choice of learning data set on the network performance are also demonstrated.

In present days, real time navigation depends on Kalman filter to fuse data from Global Positioning System (GPS) and Inertial Navigation System (INS). But there exist drawbacks like long design time, requirement of prior knowledge in fusing data from GPS and INS using Kalman Filter. In order to overcome the drawbacks of Kalman filter, GPS and INS integration are done using Artificial Neural Networks (ANN) like FFNNs. But still there exists certain drawbacks like less accuracy in FFNNs for GPS and INS data integration. Therefore, this paper introduces Higher Order Neural Networks (HONNs) approach for INS and GPS data integration to overcome the drawback. A higher order feed forward neural network architecture with optimum number of nodes is used. In the given architectures, the replacement of summation at each node by multiplication results in more powerful mapping, because of its capability of processing higher-order information from training data. Performance comparison of HONNs with FFNNs shows that the proposed architecture provides satisfactory results in terms of error rate and number of epochs. The HONNs like Multiplicative Neural Network (MNN) module and Sigma-Pi-ANN module and Traditional Feed Forward Neural Networks(FFNNs) like Radial Basis Function Neural Network (RBF) module and Back propagation Neural Network (BPN) module are trained to predict the INS position error and provide accurate position of the moving aircrafts. Sigma-Pi ANN is found to be the best in terms of accuracy, number of epochs and execution time among the two HONNs.

Ultrasound (US) imaging is the initial phase in the preliminary diagnosis for the treatment of kidney diseases, particularly to estimate kidney size, shape and position, to give information about kidney function, and to help in diagnosis of abnormalities like cysts, stones, junctional parenchyma and tumors which is shown in Figs. 7–9. This study proposes Grey Level Co-occurrence Matrix (GLCM)-based Probabilistic Principal Component Analysis (PPCA) and Artificial Neural Network (ANN) method for the classification of kidney images. Grey Wolf Optimization (GWO) is used to update the current positions of abnormal kidney images in the discrete searching space, thus getting the optimal feature subset for better classification purposes based on Feed Forward Neural Network (FFNN). The scanned image is pre-processed and the required features are extracted by GLCM, among those, some features are selected by PPCA. Feed Forward Back propagation Neural Network (FFBN) is used to classify the normalities and abnormalities in the part of kidney images. The proposed methodology is implemented in MATLAB platform and the analyzed result produces 98% accuracy using GWO-FFBN technique.