Ensembles of Biologically Inspired Optimization Algorithms for Training Multilayer Perceptron Neural Networks

Artificial neural networks (ANNs) are generally regarded to behave as black-box systems. Recent research explores various methods that can provide an insight into the internal connections and relationships existing within the network. Various methodologies that understand the input variable contribution are analyzed in detail, and rule extraction approaches for a trained artificial neural network are addressed. To understand the contribution of input variables to rainfall-runoff model combination systems, this paper for the first time investigates knowledge extraction from artificial neural network, which is used to combine the results obtained from different competing rainfall-runoff models, using three different approaches: (1) Garson’s algorithm; (2) neural interpretation diagram (NID); and (3) sensitivity analysis (SA). For the purpose of investigating knowledge extraction techniques, the trained multilayer perceptron neural network to combine the results from four different rainfall-runoff models for the Brosna Catchment located in Ireland has been chosen. The results of the three approaches obtained in this study indicate that they can be used to reduce the complexity of rainfall-runoff model combination systems by eliminating the least significant contributing input variables. Based on these approaches, the paper helps to provide guidance in the optimal number of rainfall-runoff models that best perform in a combination system.

<abstract><p>The training of artificial neural networks (ANNs) with rectified linear unit (ReLU) activation via gradient descent (GD) type optimization schemes is nowadays a common industrially relevant procedure. GD type optimization schemes can be regarded as temporal discretization methods for the gradient flow (GF) differential equations associated to the considered optimization problem and, in view of this, it seems to be a natural direction of research to <italic>first aim to develop a mathematical convergence theory for time-continuous GF differential equations</italic> and, thereafter, to aim to extend such a time-continuous convergence theory to implementable time-discrete GD type optimization methods. In this article we establish two basic results for GF differential equations in the training of fully-connected feedforward ANNs with one hidden layer and ReLU activation. In the first main result of this article we establish in the training of such ANNs under the assumption that the probability distribution of the input data of the considered supervised learning problem is absolutely continuous with a bounded density function that every GF differential equation admits for every initial value a solution which is also unique among a suitable class of solutions. In the second main result of this article we prove in the training of such ANNs under the assumption that the target function and the density function of the probability distribution of the input data are piecewise polynomial that every non-divergent GF trajectory converges with an appropriate rate of convergence to a critical point and that the risk of the non-divergent GF trajectory converges with rate 1 to the risk of the critical point. We establish this result by proving that the considered risk function is <italic>semialgebraic</italic> and, consequently, satisfies the <italic>Kurdyka-Łojasiewicz inequality</italic>, which allows us to show convergence of every non-divergent GF trajectory.</p></abstract>

BACKGROUND: Currently, artificial intelligence in the form of artificial neural networks is being actively implemented in a number of areas of our lives, including medicine. In particular, in otorhinolaryngology, artificial neural networks are used to analyze images obtained during endoscopic examinations of patients (e.g., videolaryngoscopy) [1–3]. The interpretation of laryngoscopic images often presents significant difficulties for practicing physicians, which reduces the frequency of detection of precancerous laryngeal diseases and contributes to the increase in the number of patients with stage III–IV laryngeal cancer [4, 5]. This underscores the significance of prompt performance and accurate interpretation of the findings of endoscopic examinations of patients with laryngeal disorders. Artificial neural networks can be employed to analyze the results of videolaryngoscopy, furnishing the physician with supplementary information that can enhance diagnostic accuracy and diminish the probability of error [6, 7]. AIM: The study aims to develop and train an artificial neural network for recognizing characteristic features of laryngeal neoplasms and variants of laryngeal normality. MATERIALS AND METHODS: The study was conducted under the grant of the Moscow Center for Innovative Technologies in Healthcare (grant No. 2112-1/22) entitled “Using Neural Networks (Artificial Intelligence Algorithms) for Control and Improving the Quality of Diagnosis and Treatment of Diseases of Laryngeal and Ear Structures through Digital Technologies”.The following methods were used during the course of the study: data collection for the creation of a photobank (dataset) of medical images obtained during videolaryngoscopy; data partitioning for the formation of datasets for individual nosologies and groups of diseases; the method of consilium; analysis of the accuracy of recognition and classification of digital endoscopic images; and training of classification neural networks. Consequently, a dataset comprising 1,471 laryngeal images in digital formats (JPEG, BMP) was assembled, labelled, and uploaded for the purpose of training the artificial neural network. Of the total number of images, 410 were classified as pertaining to laryngeal formation, while 1061 were classified as variants of normality. Subsequently, the neural network was trained and tested to identify the signs of normal and laryngeal masses. RESULTS: The results of the testing of the artificial neural network indicated the formation of an inaccuracy matrix, the calculation of the value of recognition accuracy, the calculation of the quality indicators of the model performance, and the construction of the ROC curve. The developed and trained artificial neural network demonstrated an accuracy of 86% in recognizing the signs of laryngeal masses and norms. CONCLUSIONS: This study demonstrates that a trained artificial neural network can successfully distinguish between signs of normal and laryngeal masses in endoscopic photographs. With further training of the neural network and achievement of high accuracy, this technology can be used in clinical practice as an assistant in the interpretation of laryngoscopic images and early diagnosis of laryngeal masses. It can also be employed to control and improve the quality of diagnosis and treatment of diseases of the throat, nose, and ears by primary care physicians.

The Sixth International Symposium on Neural Networks (ISNN 2009)

Neural networks, also called artificial neural networks (ANNs), are important techniques for performing forward/inverse modeling for microwave active/passive components to enhance a circuit design. With measured or simulated data of microwave devices, ANNs can be trained to learn relevant microwave relationships, which could be otherwise computationally expensive or for which efficient analytical formulas are not available. By training an ANN using data from electromagnetic (EM)/physics simulations, one can use the trained ANN as models for microwave circuits to replace the EM/physics models, which are typically CPU intensive, to significantly accelerate circuit design with EM/physics‐level accuracies. Fundamental concepts of the ANN structure and training, knowledge‐based neural networks, automated model generation, neuro‐transfer function modeling, deep neural network modeling, neural network‐based inverse modeling, and the use for EM/multiphysics design optimizations are described here.

Evolutionary algorithms are a class of stochastic search methods that attempts to emulate the biological process of evolution, incorporating concepts of selection, reproduction, and mutation. In recent years, there has been an increase in the use of evolutionary approaches in the training of artificial neural networks (ANNs). While evolutionary techniques for neural networks have shown to provide superior performance over conventional training approaches, the simultaneous optimization of network performance and architecture will almost always result in a slow training process due to the added algorithmic complexity. In this paper, we present a geometrical measure based on the singular value decomposition (SVD) to estimate the necessary number of neurons to be used in training a single-hidden-layer feedforward neural network (SLFN). In addition, we develop a new hybrid multiobjective evolutionary approach that includes the features of a variable length representation that allow for easy adaptation of neural networks structures, an architectural recombination procedure based on the geometrical measure that adapts the number of necessary hidden neurons and facilitates the exchange of neuronal information between candidate designs, and a microhybrid genetic algorithm ( microHGA) with an adaptive local search intensity scheme for local fine-tuning. In addition, the performances of well-known algorithms as well as the effectiveness and contributions of the proposed approach are analyzed and validated through a variety of data set types.

Rule extraction from data is one of the key technologies for solving the bottlenecks in artificial intelligence. Artificial neural networks are well suited for representing any knowledge in given data. Extraction of logical/fuzzy rules from the trained artificial neural network is of great importance to researchers in the fields of artificial intelligence and soft computing. Fuzzy rule sets are capable of approximating any nonlinear mapping relationships. Extraction of rules from data has been discussed in terms of fuzzy modeling, fuzzy clustering, and classification with fuzzy rule sets. This special issue entitled"Rule Extraction from Data" is aimed at providing the readers with good insights into the advanced studies in the field of rule extraction from data using neural networks/fuzzy rule sets. I invited seven research papers best suited for the theme of this special issue. All the papers were reviewed rigorously by two reviewers each. The first paper proposes an interesting rule extraction method from data using neural networks. Ishikawa presents a combination of learning with an immediate critic and a structural learning with forgetting. This method is capable of generating skeletal networks for logical rule extraction from data with correct and wrong answers. The proposed method is applied to rule extraction from lense data. The second paper presents a new methodology for logical rule extraction based on transformation of MLP (multilayered perceptron) to a logical network. Duck et al. applied their C-MLP2LN to the Iris benchmark classification problem as well as real-world medical data with very good results. In the third paper, Geczy and Usui propose fuzzy rule extraction from trained artificial neural networks. The proposed algorithm is implied from their theoretical study, not from heuristics. Their study enables to initially consider derivation of crisp rules from trained artificial neural network, and in case of conflict, application of fuzzy rules. The proposed algorithm is experimentally demonstrated with the Iris benchmark classification problem. The fourth paper presents a new framework for fuzzy modeling using genetic algorithm. The authors have broken new ground of fuzzy rule extraction from neural networks. For the fuzzy modeling, they have proposed a particular type of neural networks containing nodes representing membership functions. In this fourth paper, the authors discuss input variable selection for the fuzzy modeling under multiple criteria with different importance. A target system with a strong nonlinearity is used for demonstrating the proposed method. Kasabov, et al. present, in the fifth paper, a method for extraction of fuzzy rules that have different level of abstraction depending on several modifiable thresholds. Explanation quality becomes better with higher threshold values. They apply the proposed method to the Iris benchmark classification problem and to a real world problem. J. Yen and W. Gillespie address interpretability issue of Takagi-Sugeno-Kang model, one of the most popular fuzzy mdoels, in the fifth paper. They propose a new approach of fuzzy modeling that ensures not only a high approximation of the input-output relationship in the data, but also good insights about the local behavior of the model. The proposed method is applied to fuzzy modeling of sinc function and Mackey-Glass chaotic time series data. The last paper discusses fuzzy rule extraction from numerical data for high-dimensional classification problems. H.Ishibuchi, et al. have been pioneering methods for classification of data using fuzzy rules and genetic algorithm. In this last paper, they introduced a new criterion, simplicity of each rule, together with the conventional ones, compactness of rule base and classification ability, for high-dimensional problem. The Iris data is used for demonstrating their new classification method. They applied it also to wine data and credit data. I hope that the readers will be encouraged to explore the frontier to establish a new paradigm in the field of knowledge representation and rule extraction.

The dynamics of neural networks and evolutionary algorithms share common attributes and based on many research papers it seems to be that from dynamic point of view are both systems indistinguishable. In order to compare them mutually from this point of view, artificial neural networks, as similar as possible to natural one, are needed. In this paper is described part of our research that is focused on the synthesis of artificial neural networks. Since most current ANN structures are not common in nature, we introduce a method of a complex network synthesis using network growth model, considered as a neural network. Synaptic weights of the synthesized ANN are then trained by an evolutionary algorithm to respond to an input training set successfully.

In this paper, we show the possibility of creating and identifying the features of an artificial neural network (ANN), which consists of mathematical models of biological neurons. The FitzHugh–Nagumo (FHN) system is used as a paradigmatic model demonstrating basic neuron activities. First, in order to reveal how biological neurons can be embedded within an ANN, we train the ANN with nonlinear neurons to solve a basic image recognition problem with an MNIST database; next, we describe how FHN systems can be introduced into this trained ANN. After all, we show that an ANN with FHN systems inside can be successfully trained with improved accuracy comparing with first trained ANN and then with inserted FHN systems. This approach opens up great opportunities in terms of the direction of analog neural networks, in which artificial neurons can be replaced by more appropriate biological ones.

Using Orthogonal Grey Wolf Optimizer with Mutation for Training Multi-Layer Perceptron Neural Network

A scheme to predict transient stability status following a disturbance is presented. The scheme is activated upon the tripping of a line or bus and operates as follows: Samples of frequency deviations of all generator buses are obtained using a sampling frequency of 32 samples per cycle. For each generator bus, the maximum frequency deviation within the first two samples is obtained. A vector is then constructed from the obtained maximum frequency deviations. The Euclidean norm of the vector is calculated and then fed as input to a trained multilayer perceptron neural network which predicts the stability status of the system. The scheme was tested using the New England test system. The prediction accuracy was found to be 99.49%.

This article, written by Technology Editor Dennis Denney, contains highlights of paper SPE 94357, "Treating Uncertainties in Reservoir-Performance Prediction With Neural Networks," by J.P. Lechner, SPE, OMV A.G., and G. Zangl, SPE, Schlumberger Information Solutions, prepared for the 2005 SPE Europec/EAGE Annual Conference, Madrid, Spain, 13–16 June. In development projects, reservoir parameters are known only within certain ranges, which results in various realizations of the subsurface. Because of the computational time involved, simulation models to obtain a probability distribution of possible outcomes cannot cover all possible parameter combinations. Creating a response surface based on a reduced number of simulation runs becomes necessary. Such a response surface can be used to approximate results for several variations of input parameters. An approach in which reservoir response is captured by an artificial neural network (ANN) has been investigated. The trained ANN model was used in Monte Carlo simulations to generate the probability distribution of possible outcomes. Introduction Often, simulation models are built to design field-development plans. Because of the low level of information during the early development stage, a range of uncertainty in the prediction scenarios must be considered. Such uncertainties can be handled with ANNs. Numerical models (simulation models) are detailed and powerful predictive tools that can answer many questions regarding subsurface engineering. However, especially in the early development phase, uncertainties are large, and prediction results will span a broad range. To take into account the whole range of possible outcomes of reservoir simulation, optimization routines would have to run the numerical models perhaps thousands of times. Although the most likely solution that respects subsurface uncertainties is of interest, the most profitable solution for reservoir management must be found. Because of the computational time involved, these methodologies often are not used. Building a response surface that can predict many possible outcomes of a numerical simulation by processing a large variety of input parameters could provide a solution to this problem. The response surface must cover the non-linear dependencies between input and output parameters. Experimental design and response-surface modeling has been used to delineate subsurface uncertainty. Common applications include determining the probability distribution of original oil in place from geological models, quantifying uncertainty in predevelopment or preredevelopment projects, predicting production performance from different realizations of a geological model, and optimizing the location of new wells to maximize net profit. Methodology Fig. 1 shows the proposed workflow. As a first step, a limited number of simulation runs is carried out to define the most sensitive input parameters. In these runs, parameters are altered one at a time, either to the minimum or to the maximum. After selecting the most influential input parameters, three values (minimum, best-estimate, and maximum) are defined, and an experimental design is set up. The experimental design aims at obtaining the maximum information with a minimum number of simulation runs. However, practice showed that a minimum number of training data sets are required by the ANN to be able to make sufficiently accurate predictions. With these data sets containing varying input parameters and the corresponding output, an ANN is trained and tested with several additional simulation runs. After defining the probability distribution of the input parameters, the trained ANN is used as a proxy function for a Monte Carlo simulation. Thereby, a probability distribution of the desired output parameter (e.g., cumulative production after a certain number of years) is gained. The quality of the ANN is measured by the root-mean-square error of the deviation between the predicted and the actual output.

The existence of various sounds from different natural and unnatural sources in the deep sea has caused the classification and identification of marine mammals intending to identify different endangered species to become one of the topics of interest for researchers and activist fields. In this paper, first, an experimental data set was created using a designed scenario. The whale optimization algorithm (WOA) is then used to train the multilayer perceptron neural network (MLP-NN). However, due to the large size of the data, the algorithm has not determined a clear boundary between the exploration and extraction phases. Next, to support this shortcoming, the fuzzy inference is used as a new approach to developing and upgrading WOA called FWOA. Fuzzy inference by setting FWOA control parameters can well define the boundary between the two phases of exploration and extraction. To measure the performance of the designed categorizer, in addition to using it to categorize benchmark datasets, five benchmarking algorithms CVOA, WOA, ChOA, BWO, and PGO were also used for MLPNN training. The measured criteria are concurrency speed, ability to avoid local optimization, and the classification rate. The simulation results on the obtained data set showed that, respectively, the classification rate in MLPFWOA, MLP-CVOA, MLP-WOA, MLP-ChOA, MLP-BWO, and MLP-PGO classifiers is equal to 94.98, 92.80, 91.34, 90.24, 89.04, and 88.10. As a result, MLP-FWOA performed better than other algorithms.

In this work a modification was made on the algorithm of Artificial Neural Networks (NN) Training of the Multilayer Perceptron type (MLP) based on multi-objective optimization (MOBJ), to increase its computational efficiency. Usually, the number of efficient solutions to be generated is a parameter that must be provided by the user. In this work, this number is automatically determined by an algorithm, through the usage of golden section, being generally less when specified, showing a sensible reduction in the processing time and keeping the high generalization capability of the obtained solution from the original method.

The conventional technique for Artificial Neural Network (ANN) training, that is, the Back Propagation algorithm (BP), require large number of iterations and high computational time. The induction of meta-heuristic algorithms improve the training process by introducing randomness into the architecture. However even these algorithms are inclined to fall into the local minima when cost solution search spaces are at greater dimensions and/or convergence rate is slow. To address these inefficiencies in an artificial neural network, the manuscript presents a novel ANN classifier that updates its weights and biases using the simulation of the improved hybrid grey wolf sine cosine algorithm (IHGWOSCA). The proposed network, improved hybrid grey wolf sine cosine algorithm based neural network, consists of an input and output layer and a fully-connected single hidden layer of 10 neurons. Three University of California Irvine (UCI) database datasets were used as basis for the benchmark test namely; ‘Cryotherapy’, ‘Haberman’ and ‘Blood Transfusion Service Center’, training accuracies for which are 93.33%, 74.01% and 76.75% whereas the testing accuracies are 93.33%, 76.47% and 81.52% respectively. Comparative analysis with Group Teaching Optimization Algorithm-Neural Network (GTOANN), Particle Swarm Optimization-Neural Network (PSONN) and Hybrid Grey Wolf Sine Cosine Optimization Algorithm Neural Network (HGWOSCA-NN) show that the proposed IHGWOSCA-NN outperforms in training accuracies as well as in testing accuracies.