Algebraic Zero Error Training Method for Neural Networks Achieving Least Upper Bounds on Neurons and Layers

The paper is focused on the idea to demonstrate the advantages of deep learning approaches over ordinary shallow neural network on their comparative applications to image classifying from such popular benchmark databases as FERET and MNIST. An autoassociative neural network is used as a standalone program realized the nonlinear principal component analysis for prior extracting the most informative features of input data for neural networks to be compared further as classifiers. A special study of the optimal choice of activation function and the normalization transformation of input data allows to improve efficiency of the autoassociative program. One more study devoted to denoising properties of this program demonstrates its high efficiency even on noisy data. Three types of neural networks are compared: feed-forward neural net with one hidden layer, deep network with several hidden layers and deep belief network with several pretraining layers realized restricted Boltzmann machine. The number of hidden layer and the number of hidden neurons in them were chosen by cross-validation procedure to keep balance between number of layers and hidden neurons and classification efficiency. Results of our comparative study demonstrate the undoubted advantage of deep networks, as well as denoising power of autoencoders. In our work we use both multiprocessor graphic card and cloud services to speed up our calculations. The paper is oriented to specialists in concrete fields of scientific or experimental applications, who have already some knowledge about artificial neural networks, probability theory and numerical methods.

This paper presents a fully complex-valued relaxation network (FCRN) with its projection-based learning algorithm. The FCRN is a single hidden layer network with a Gaussian-like sech activation function in the hidden layer and an exponential activation function in the output layer. For a given number of hidden neurons, the input weights are assigned randomly and the output weights are estimated by minimizing a nonlinear logarithmic function (called as an energy function) which explicitly contains both the magnitude and phase errors. A projection-based learning algorithm determines the optimal output weights corresponding to the minima of the energy function by converting the nonlinear programming problem into that of solving a set of simultaneous linear algebraic equations. The resultant FCRN approximates the desired output more accurately with a lower computational effort. The classification ability of FCRN is evaluated using a set of real-valued benchmark classification problems from the University of California, Irvine machine learning repository. Here, a circular transformation is used to transform the real-valued input features to the complex domain. Next, the FCRN is used to solve three practical problems: a quadrature amplitude modulation channel equalization, an adaptive beamforming, and a mammogram classification. Performance results from this paper clearly indicate the superior classification/approximation performance of the FCRN.

In this work, a new adaptive extreme learning machine (ELM) neural network-based fuzzy controller is designed and simulated for implementing speed regulation in a permanent magnet synchronous motor. ELM is a neural model wherein the number of hidden neurons to be placed in the hidden layer is tuned during the process of neural network training itself. A new adaptive ELM model is developed for placing the number of hidden neurons in the hidden layer, and this new adaptive ELM is tuned with artificial bee colony (ABC) algorithm for optimizing its weight parameters and also the number of hidden neurons. Fuzzy proportional–integral (PI) controller is developed in this work in order to eliminate the steady-state error. The new adaptive ELM neural model optimized with ABC algorithm is applied to tune the input parameters of the fuzzy PI controller and also on optimizing the rules and fuzzy membership functions. The optimized adaptive ELM neural network-based fuzzy PI controller is utilized to investigate the speed regulation of permanent magnet synchronous motor (PMSM) in this work. The developed new PI controller with the PMSM is tested for its performance characteristics and to prove its validity is compared with the traditional controller and other heuristic controllers proposed in earlier literature works.

Orthogonal transformations, proper decomposition, and the Moore–Penrose inverse are traditional methods of obtaining the output layer weights for an extreme learning machine autoencoder. However, an increase in the number of hidden neurons causes higher convergence times and computational complexity, whereas the generalization capability is low when the number of neurons is small. One way to address this issue is to use the fast iterative shrinkage-thresholding algorithm (FISTA) to minimize the output weights of the extreme learning machine. In this work, we aim to improve the convergence speed of FISTA by using two fast algorithms of the shrinkage-thresholding class, called greedy FISTA (G-FISTA) and linearly convergent FISTA (LC-FISTA). Our method is an exciting proposal for decision-making involving the resolution of many application problems, especially those requiring longer computational times. In our experiments, we adopt six public datasets that are frequently used in machine learning: MNIST, NORB, CIFAR10, UMist, Caltech256, and Stanford Cars. We apply several metrics to evaluate the performance of our method, and the object of comparison is the FISTA algorithm due to its popularity for neural network training. The experimental results show that G-FISTA and LC-FISTA achieve higher convergence speeds in the autoencoder training process; for example, in the Stanford Cars dataset, G-FISTA and LC-FISTA are faster than FISTA by 48.42% and 47.32%, respectively. Overall, all three algorithms maintain good values of the performance metrics on all databases.

This paper proposes a step-by-step technique for combining basic models that forecast electricity consumption in an artificial neural network by the method of preliminary selection and further hybridization. The reported experiments were conducted using data on hourly electricity consumption at the metallurgical plant AO ArcelorMittal Temirtau in the period from January 1, 2019, to November 30, 2021. The current research is related to the planned introduction of a balancing electricity market. 96 combinations of basic models were compiled, differing in the type of neural network, the set of initial data, the order of lag, the learning algorithm, and the number of neurons in the hidden layer. It has been determined that the NARX-type network is the most optimal architecture to forecast electricity consumption. Based on experimental studies, the number of hidden neurons needed to form a planned daily profile should equal 3 or 4; it is recommended to use the conjugate gradient method as a learning algorithm. When selecting models from three groups, it was revealed that the conjugate gradient method produces better results compared to the Levenberg-Marquardt algorithm. It is determined that the values of the selected RMSE error indicator take values of 23.17, 22.54, and 22.56, respectively, for the first, second, and third data groups. The adaptive hybridization method has been shown to reduce the RMSE error rate to 21.73. However, the weights of the best models with values of 0.327 for the first group of data, and 0.336 for the second and third ones, show that the individual use of a separate combination of models is also applicable. The devised forecasting electricity consumption model can be integrated into an automated electricity metering system

Assessment of TiO2 band gap from structural parameters using artificial neural networks

Electrical load forecasting is one of the important parts for smart grid system. The reliable prediction of the load demand contributes to the efficient and economical operations and planning. The artificial neural network is used extensively in load demand forecasting. The nonlinear nature of the electrical load demand conforms to the ability of the artificial neural network in calculating the nonlinear relationship of inputs and outputs. Among many models of neural networks, radial basis neural networks yield superior performance in small error and fast simulation time. However, it is challenge to design the radial basis neural networks. The excessive numbers of hidden neurons lead to lacking of generalization or so called overfitting problems. This paper proposes an approach to design the radial basis neural networks that use as least numbers of hidden neurons as possible. The error criterion is optimized based on modified genetic algorithm as the numbers of hidden neurons are incrementally increased. Simulation results of short term load forecasting are calculated in Matlab, and compared to the orthogonal least square error method. The proposed approach gives better results with the same numbers of hidden neurons.

On extreme learning machines in sequential and time series prediction: A non-iterative and approximate training algorithm for recurrent neural networks

The efficiency of the backpropagation algorithm is considered in relation to the problem of training a three-layer perceptron with two input and two output values. To activate perceptron neurons, a logistic function with parameter a was used. The learning process was carried out using many of 10,000 examples and lasted for 100 epochs. Two training modes are considered - deterministic, in which the sequence of training examples in each epoch is the same, and stochastic, when the training examples are randomly reordered before each epoch. Based on the simulation results, the advantages of the deterministic mode over the stochastic mode were not revealed, which makes it possible to recommend deterministic learning for practical use as it is simpler in computational terms. It is shown that the best training quality is achieved when using 8 neurons in the second hidden layer, when the average relative error of the trained perceptron is less than 0.003. With a decrease in the number of hidden neurons, the quality of learning decreases, with an increase, the range of learning parameters narrows, at which the learning process converges to a satisfactory small error, while the likelihood of an imbalance in synaptic weights increases. With the learning rate parameter values of 0.95, the best quality indicators and the minimum duration of their oscillations between epochs were obtained; a higher rate of convergence of the learning process was recorded. The influence of the parameter a on the quality of training is investigated for its values up to 3.375 inclusive. When the size of the second hidden layer is no more than 8 neurons, an increase in a leads to a decrease in the average error of the trained perceptron. In perceptrons with 16 or more hidden neurons, the convergence region of the algorithm shifts towards small a values (no more than 1.125 for 32 neurons).

Self-scaled conjugate gradient training algorithms

A new model selection algorithm is established to determine the best number of hidden neurons for radial basis function neural networks. We used a Bayesian information-based criterion to select the best number of hidden neurons. The new algorithm grows the number of hidden neurons while the Bayesian information-based criterion is used for improvement. The optimal parameter values of a current neural network are used in the subsequent architecture. The computational results are compared with the trial-and-error approach through publicly available data sets. It is found that the new algorithm is suitable to improve the performance of the neural networks automatically. The root mean square error function is used to measure out-of-sample performance.

Nonlinear System Identification Using Multilayer Neural Networks: Some Ideas for Initial Weights, Number of Hidden Neurons and Error Criteria

In this work, a new training algorithm for probabilistic neural networks (PNN) is presented. The proposed algorithm addresses one of the major drawbacks of probabilistic neural networks, which is the size of the hidden layer in the network. By using a cross-validation training algorithm, the number of hidden neurons is shrunk to a smaller number consisting of the most representative samples of the training set. This is done without affecting the overall architecture of the network. Performance of the new network is compared against performance of standard probabilistic neural networks for different databases from the UCI database repository. Results show an important gain in network size and performance.

The aim of data transformation is to transform the original feature space of data into another space with better properties. This is typically combined with dimensionality reduction, so that the dimensionality of the transformed space is smaller. A widely used method for data transformation and dimensionality reduction is Principal Component Analysis (PCA). PCA finds a subspace that explains most of the data variance. While the new PCA feature space has interesting properties, such as removing linear correlation, PCA is an unsupervised method. Therefore, there is no guarantee that the PCA feature space will be the most appropriate for supervised tasks, such as classification or regression. On the other hand, 3-layer Multi Layer Perceptrons (MLP), which are supervised methods, can also be understood as a data transformation carried out by the hidden layer, followed by a classification/regression operation performed by the output layer. Given that the hidden layer is obtained after a supervised training process, it can be considered that it is performing a supervised data transformation. And if the number of hidden neurons is smaller than the input, also dimensionality reduction. Despite this kind of transformation being widely available (any neural network package that allows access to the hidden layer weights can be used), no extensive experimentation on the quality of 3-layer MLP data transformation has been carried out. The aim of this article is to carry out this research for classification problems. Results show that, overall, this transformation offers better results than the PCA unsupervised transformation method.

Ovarian masses are common and a good pre-surgical assessment of their nature is important for adequate treatment. Bayesian Multi-Layer Perceptrons (MLPs) using the evidence procedure were used to predict whether tumors are malignant or not. Automatic Relevance Determination (ARD) is used to select the most relevant of the 40+ available variables. Cross-validation is used to select an optimal combination of input set and number of hidden neurons. The data set consists of 1066 tumors collected at nine centers across Europe. Results indicate good performance of the models with AUC values of 0.93-0.94 on independent data. A comparison with a Bayesian perceptron model shows that the present problem is to a large extent linearly separable. The analyses further show that the number of hidden neurons specified in the ARD analyses for input selection may influence model performance.