An implementation of the Levenberg–Marquardt algorithm for simultaneous-energy-gradient fitting using two-layer feed-forward neural networks

The BP (back-propagation algorithm) and GA (genetic algorithm) are among the most effective algorithms of neural networks (NN). As a deterministic gradient-descent algorithm and a stochastic optimizing algorithm respectively, there exits great compensability between their advantages and disadvantages. The proposed hybrid BP/GA learning method for a multilayer feedforward net blends the merits of both the BP and GA. Based on BP/GA, a two-layer feedforward neural network is designed. HSPICE simulation results have proved its ability to solve the XOR problem.

In order to solve the problem that the traditional long-term high-speed traffic forecasting algorithm is affected by the approximation ability of the function and easy to fall into the local mass value, we wrote a multivariate-based highway traffic forecasting algorithm scaling and convolutional networks. Because the feedforward wavelet neural network algorithm predicts the short-term traffic flow in different areas, it is necessary to examine the ability to predict the difference between different models. From the standard feedforward wavelet neural network algorithm using global optimization capabilities, we improve the wolf pack algorithm, improve the search accuracy of the algorithm, get the best solution of the estimated value of the work according to the search results when completing the research objectives, and get the ability to predict the work of the model. Feedforward neural network algorithm: we develop and obtain the best short-term high-speed traffic forecast values. The results are as follows: after using the author’s algorithm, the processing time increases by 1.5 seconds, but the average percentage of errors decreases by more than 50%, in fact the error and the root mean square error decreased by about 30%, and the smoothing coefficient increased by about 1%. The prediction of the author’s algorithm for short-term high-speed traffic is better than the wavelet neural network prediction algorithm, and the prediction accuracy and stability of the author’s algorithm are higher.

A fast and robust algorithm for training feedforward neural networks (FNNs) by using a variable forgetting factor and U-D factorization-based recursive prediction error (RPE) method is proposed. In comparison with the backpropagation (BP) and RPE based learning algorithms, the proposed algorithm, called UD-RPE, can provide much more accurate learning results in fewer iterations with fewer hidden nodes and improve convergence rate and numerical stability (robustness). In addition, it is less sensitive to start-up parameters, such as initial weights and initial covariance matrix, and the randomness in the observed data. It also has good generalization ability and needs less learning time. Simulation results of nonlinear dynamic system modeling and identification show that the algorithm proposed here is an effective and efficient learning algorithm for FNNs.

Modelling and Optimizing Pyrene Removal from the Soil by Phytoremediation using Response Surface Methodology, Artificial Neural Networks, and Genetic Algorithm

Many real scenarios in machine learning are non-stationary. These challenges forces to develop new algorithms that are able to deal with changes in the underlying problem to be learnt. These changes can be gradual or abrupt. As the dynamics of the changes can be different, the existing machine learning algorithms exhibit difficulties to cope with them. In this work we propose a new method, that is based in the introduction of a forgetting function in an incremental online learning algorithm for two-layer feedforward neural networks. This forgetting function gives a monotonically crescent importance to new data. Due to this fact, the network forgets in presence of changes while maintaining a stable behavior when the context is stationary. The theoretical basis for the method is given and its performance is illustrated by evaluating its behavior. The results confirm that the proposed method is able to work in evolving environments.

The unsteady recovery of oil and gas prices in early 2017 led to an increase in drilling and hydraulic fracturing operations in North America liquid-rich shale plays. The increasing number of producing wells, in addition to re-fractured wells, impose the need for optimizing field development strategies and shale gas supply chain networks that maximize profitability. Moreover, operators must account for undulating natural gas demands both locally and externally in a persistently low oil price environment. In this paper, we adopt supervised machine learning approaches to forecast local natural gas demand as well as guide well re-frac candidature. Both serve as critical inputs for maximizing the net present value of any shale gas field development project. An optimized water management structure is also incorporated in the proposed framework to account for associated produced-water recycling. Considering a shale gas network in the Marcellus play, supply chain optimization was achieved using a mixed integer non-linear programming formulation. The Strategic Planning model relies on at least 4 major efforts including reservoir simulation, which in turn relies on output from a feed forward Neural Network (NN) algorithm. The trained NN algorithm was deemed suitable for recommending re-frac candidates, necessary decision variables for multiphase reservoir simulation. Finally, NPV optimization relied on a four-layer Long Short-Term Memory (LSTM) recurrent neural network, developed for forecasting local shale gas demand. Both neural network algorithms were scripted using python. 17 non-redundant parameters were mined for 250 wells in the case study. The t-Distributed Stochastic Neighbor Embedding technique was used to visualize related low dimensional manifolds within the high-dimensional data set, and a NN algorithm was used to obtain probabilities of misclassification. The well-pads in the case study superstructure are modeled using compositional reservoir simulation. Each reservoir model is tuned by history matching production and pressure data for designated producing wells. Alternative field development strategies (including re-fracturing) are then simulated for a 10-year planning horizon to generate gas and water rate decline profiles. Using the LSTM developed, local gas demand is forecasted using data sets created from multivariate time-dependent local and global variables affecting shale gas demand. The LSTM algorithm is derived by convolving some features from the raw data set, with fined-tuned weights, with the objective of minimizing a pre-defined demand error function. The results obtained show that application of this integrated approach can give operators a calculated advantage, preventing erroneous feedback of project profitability but allowing early time decision-making that maximizes shale asset NPV.

Machine learning approaches have been promising in constructing high-dimensional potential energy surfaces (PESs) for molecules and materials. Neural networks (NNs) are one of the most popular such tools because of its simplicity and efficiency. The training algorithm for NNs becomes essential to achieve a fast and accurate fit with numerous data. The Levenberg-Marquardt (LM) algorithm has been recognized as one of the fastest and robust algorithms to train medium sized NNs and widely applied in recent NN based high quality PESs. However, when the number of ab initio data becomes large, the efficiency of LM is limited, making the training time consuming. Extreme learning machine (ELM) is a recently proposed algorithm which determines the weights and biases of a single hidden layer NN by a linear solution and is thus extremely fast. It, however, does not produce sufficiently small fitting error because of its random nature. Taking advantages of both algorithms, we report a generalized hybrid algorithm in training multilayer NNs. Tests on H+H2 and CH4+Ni(111) systems demonstrate the much higher efficiency of this hybrid algorithm (ELM-LM) over the original LM. We expect that ELM-LM will find its widespread applications in building up high-dimensional NN based PESs.

In this paper a general class of fast learning algorithms for feedforward neural networks is introduced and described. The approach exploits the separability of each layer into linear and nonlinear blocks and consists of two steps. The first step is the descent of the error functional in the space of the outputs of the linear blocks (descent in the neuron space), which can be performed using any preferred optimization strategy. In the second step, each linear block is optimized separately by using a least squares (LS) criterion. To demonstrate the effectiveness of the new approach, a detailed treatment of a gradient descent in the neuron space is conducted. The main properties of this approach are the higher speed of convergence with respect to methods that employ an ordinary gradient descent in the weight space backpropagation (BP), better numerical conditioning, and lower computational cost compared to techniques based on the Hessian matrix. The numerical stability is assured by the use of robust LS linear system solvers, operating directly on the input data of each layer. Experimental results obtained in three problems are described, which confirm the effectiveness of the new method.

The convergence of a class of stochastic pseudo-gradient algorithms driven by correlated data sequences is considered in this paper. The obtained results are applied to a learning algorithm for feedforward neural networks and sufficient conditions for its convergence are determined.

The reproducing kernel Hilbert space (RKHS) method has been previously shown to be accurate and efficient in the construction of potential energy surfaces (PES) by interpolating fully gridded high level ab initio data. This paper extends the RKHS method to handle partially filled data calculated over arbitrarily shaped regions, while keeping nearly intact its accuracy and efficiency. The extension permits points or regions to be added to or removed from the grid as needed before doing expensive ab initio calculations, thus enabling the construction of RKHS PESs from the data distributions that are most likely to occur in practice. The utility of the new technique is demonstrated using data from the lowest global RKHS PES for the reaction O(1D)+H2, showing that ignoring the irrelevant regions of the PES does not adversely impact the accuracy of the surfaces if the relevant region is adequately sampled.

The output weight optimization-hidden weight optimization (OWO-HWO) feedforward network training algorithm alternately solves linear equations for output weights and reduces a separate hidden layer error function with respect to hidden layer weights. Here, a new hidden layer error function is proposed which de-emphasizes net function errors that correspond to saturated activation function values. In addition, an adaptive learning rate based on the local shape of the error surface is used in hidden layer training. Faster learning convergence is experimentally verified.

The worldwide increasing applications of nonlinear loads, mostly consisting of power electronics devices, have made the power quality problems an important concern than ever before for utilities and consumers. Therefore, detection and classification of Power Quality disturbances is highly desirable. Most power quality disturbances are non-stationary and transitory and their detection and classification have proved to be much needed. Hence there is a requirement of advanced tools and techniques for the analysis of Power Quality disturbances. In this work detection and classification of various types of transients and interruptions, caused due to severe fault and reclosing of circuit breaker, is done by using Discrete Wavelet Transform and two-layer feed forward neural network. This work presents new approach aimed at automating the analysis of power quality disturbances including transients, interruption and normal waveform. The disturbance current waveform can be obtained from the disturbance generation model. The Discrete Wavelet Transform is chosen for feature extraction. Feature extraction outputs are the coefficients (detailed and approximate) of Discrete Wavelet Transform represents the power quality disturbance signal at the different levels in time and frequency domain.

Given the compressive strength of concrete, yield strength of steel, span, dead and live loads, singly reinforced concrete simple beams are first optimally designed using genetic algorithms with constraints satisfying the specifications of the ACI code. The objective function is to minimize the total cost of tension steels, stirrups and concrete. A variety of beams are designed for the use of the neural network. To train and test the effectiveness of the neural network, these optimal results are randomly divided into three sets: the training set, validation set and test set. This paper uses a two-layer feed forward neural network: one hidden layer and one output layer. The transfer functions for the hidden layer and output layer are tan-sigmoid and linear functions, respectively. The inputs of the neural network are the compressive strength of concrete, yield strength of steel and span, width and effective depth of the beam, as well as vertical loads the beam is subjected to; the targets of the neural network are the steel ratio and cost of the beam. To evaluate the accuracy, the regression analysis of the target and network output is carried out. Numerical results show good performance of the neural network, which can be used as a model to predict the lowest cost and steel ratio of singly reinforced concrete simple beams. Key words: Reinforced concrete beams, genetic algorithm, neural network, regression analysis.

The effective short-term forecasting of wind speeds for wind farm applications requires the tailored application of a collection of various forecasting tools. Physical forecasting methods generally form the basis of these forecasting tools, coupled with regression and statistical methods. In this work, the Weather Research and Forecasting (WRF) model is utilized to generate an ensemble spread for a targeted site. The optimal deterministic forecast is subsequently determined using two techniques: a novel application of the mean-variance portfolio theorem; as well as a two-layered feed forward neural network. The mean-variance portfolio optimization technique is assessed for its suitability for weighted ensemble selection by comparing its optimized deterministic forecast accuracy with a neural network approach.

The thermo-physical properties of the ionic liquids are required for engineering and product design applications in many pharmaceutical and food industries. Among the many unique properties of ionic liquids, their surface tension plays an important role for various reasons. In the present study, the aim is to investigate the effect of temperature and concentration on the surface tension of the binary solution of 1-butyl-3-methylimidazolium bromide + water. The concentration of the ionic liquid is varied from 0.1–0.6%w/w and temperature ranges from 302.85–337.45 K. A quadratic mathematical model has been formulated for predicting the surface tension using response surface methodology with central composite rotatable design having a coefficient of determination R 2 as 0.9807. In addition, a two-layered feed forward back propagation neural network 2-4-1 is also modelled which provides a better performance when compared to response surface model. The developed ANN model can predict the surface tension with mean square error, root mean square error and percentage absolute average error equal to 0.156, 0.395 and 0.623, respectively.