Adjusting Weights in Artificial Neural Networks using Evolutionary Algorithms

A comparative study of hybrid estimation distribution algorithms in solving the facility layout problem

Estimation of distribution algorithms (EDAs) are a successful branch of evolutionary algorithms (EAs) that evolve a probabilistic model instead of a population. Analogous to genetic drift in EAs, EDAs also encounter the phenomenon that the random sampling in the model update can move the sampling frequencies to boundary values not justified by the fitness. This can result in a considerable performance loss. This work gives the first tight quantification of this effect for three EDAs and one ant colony optimizer, namely for the univariate marginal distribution algorithm, the compact genetic algorithm, population-based incremental learning, and the max-min ant system with iteration-best update. Our results allow to choose the parameters of these algorithms in such a way that within a desired runtime, no sampling frequency approaches the boundary values without a clear indication from the objective function. This paper for the Hot-off-the-Press track at GECCO 2020 summarizes the work Sharp Bounds for Genetic Drift in Estimation of Distribution Algorithms by B. Doerr and W. Zheng, which has been accepted for publication in the IEEE Transactions on Evolutionary Computation [5].

The Influence of Training Data Selection on Performance of Neural Networks for Control of Non-Linear Systems *

The influence of training data selection on performance of neural networks for control of non-linear systems

Evolutionary algorithms and elliptical copulas applied to continuous optimization problems

An improved genetic algorithm is proposed in which a diffusing operator is designed. Gaussian mutation method is applied in diffusing operator and its task is mainly to perform local search. Connection weights of an artificial neural network are trained on standard XOR problem by using the proposed genetic algorithm. The results show that the proposed genetic algorithm can perform both global search and local search efficiently, therefore, it can be used to train artificial neural networks alone rather than incorporate other local search algorithms, such as BP to improve local search of training algorithm, so the proposed genetic algorithm is significant to simplify training algorithm of artificial neural networks and improve training efficiency

<abstract><p>The training of artificial neural networks (ANNs) with rectified linear unit (ReLU) activation via gradient descent (GD) type optimization schemes is nowadays a common industrially relevant procedure. GD type optimization schemes can be regarded as temporal discretization methods for the gradient flow (GF) differential equations associated to the considered optimization problem and, in view of this, it seems to be a natural direction of research to <italic>first aim to develop a mathematical convergence theory for time-continuous GF differential equations</italic> and, thereafter, to aim to extend such a time-continuous convergence theory to implementable time-discrete GD type optimization methods. In this article we establish two basic results for GF differential equations in the training of fully-connected feedforward ANNs with one hidden layer and ReLU activation. In the first main result of this article we establish in the training of such ANNs under the assumption that the probability distribution of the input data of the considered supervised learning problem is absolutely continuous with a bounded density function that every GF differential equation admits for every initial value a solution which is also unique among a suitable class of solutions. In the second main result of this article we prove in the training of such ANNs under the assumption that the target function and the density function of the probability distribution of the input data are piecewise polynomial that every non-divergent GF trajectory converges with an appropriate rate of convergence to a critical point and that the risk of the non-divergent GF trajectory converges with rate 1 to the risk of the critical point. We establish this result by proving that the considered risk function is <italic>semialgebraic</italic> and, consequently, satisfies the <italic>Kurdyka-Łojasiewicz inequality</italic>, which allows us to show convergence of every non-divergent GF trajectory.</p></abstract>

Artificial neural networks (ANN) and evolutionary algorithms are two relatively young research areas that were subject to a steadily growing interest during the past years. This paper examines the use of different evolutionary algorithms, imperialist competitive algorithm (ICA), genetic algorithm (GA), ICA-GA and recursive ICA-GA (R-ICA-GA) to train a classification problem on a multi layer perceptron (MLP) neural network. All of named evolutionary training algorithms are compared together in this paper. The first goal of the paper is to apply new evolutionary optimization algorithms ICA-GA and R-ICA-GA for training the ANN and the second goal of the paper is to compare different evolutionary algorithms. It is shown that the ICA-GA has the best performance, in number of epochs, compared to the other algorithms. For this purpose, learning algorithms are applied on six known datasets (WINE, PIMA, WDBC, IRIS, SONAR and GLASS) which are used for classification problems.

This paper introduces the genetic algorithm (GA) into the artificial neural network (ANN) for the parametric modeling of ultra-wideband bandpass filters. With the optimization of thresholds and weights in ANN, GA speeds up the convergence of ANN training. The trained ANN, instead of the time-consuming full-wave simulation, is repeatedly called in the design process, leading to an efficient solution.

Estimation of distribution algorithm (EDA) is a new class of evolutionary algorithms with a wide range of real-world applications. However, it has been well known that the performance of EDA is not satisfactory enough if its population size is small. But to simply increase its population size may result in slow convergence. To the best knowledge of the authors', very few work has been done on improving the performance of EDA under small population size. This paper illustrates why EDA does not work well under small population size and proposes a novel approach termed as Over-Selection to boost EDA under small population size. Experimental results on several benchmark problems demonstrate that Over-Selection based EDA is often able to achieve a better solution without significantly increasing its time consumption when compared with the original version of EDA.

Making Artificial Brains: Components, Topology, and Optimization.

The emergence of different metaheuristics and their new variants in recent years has made the definition of the term Evolutionary Algorithms unclear. Originally, it was coined to put a group of stochastic search algorithms that mimic natural evolution together. While some people would still see it as a specific term devoted to this group of algorithms, including Genetic Algorithms, Genetic Programming, Evolution Strategies, Evolutionary Programming, and to a lesser extent Differential Evolution and Estimation of Distribution Algorithms, many others would regard “Evolutionary Algorithms” as a general term describing population-based search methods that involve some form of randomness and selection. In this chapter, we re-visit the fundamental question of “what is an Evolutionary Algorithm?” not only from the traditional viewpoint but also the wider, more modern perspectives relating it to other areas of Evolutionary Computation. To do so, apart from discussing the main characteristics of this family of algorithms we also look at Memetic Algorithms and the Swarm Intelligence algorithms. From our discussion, we see that establishing semantic borders between these algorithm families is not always easy, nor necessarily useful. It is anticipated that they will further converge as the research from these areas cross-fertilizes each other.

Rule extraction from data is one of the key technologies for solving the bottlenecks in artificial intelligence. Artificial neural networks are well suited for representing any knowledge in given data. Extraction of logical/fuzzy rules from the trained artificial neural network is of great importance to researchers in the fields of artificial intelligence and soft computing. Fuzzy rule sets are capable of approximating any nonlinear mapping relationships. Extraction of rules from data has been discussed in terms of fuzzy modeling, fuzzy clustering, and classification with fuzzy rule sets. This special issue entitled"Rule Extraction from Data" is aimed at providing the readers with good insights into the advanced studies in the field of rule extraction from data using neural networks/fuzzy rule sets. I invited seven research papers best suited for the theme of this special issue. All the papers were reviewed rigorously by two reviewers each. The first paper proposes an interesting rule extraction method from data using neural networks. Ishikawa presents a combination of learning with an immediate critic and a structural learning with forgetting. This method is capable of generating skeletal networks for logical rule extraction from data with correct and wrong answers. The proposed method is applied to rule extraction from lense data. The second paper presents a new methodology for logical rule extraction based on transformation of MLP (multilayered perceptron) to a logical network. Duck et al. applied their C-MLP2LN to the Iris benchmark classification problem as well as real-world medical data with very good results. In the third paper, Geczy and Usui propose fuzzy rule extraction from trained artificial neural networks. The proposed algorithm is implied from their theoretical study, not from heuristics. Their study enables to initially consider derivation of crisp rules from trained artificial neural network, and in case of conflict, application of fuzzy rules. The proposed algorithm is experimentally demonstrated with the Iris benchmark classification problem. The fourth paper presents a new framework for fuzzy modeling using genetic algorithm. The authors have broken new ground of fuzzy rule extraction from neural networks. For the fuzzy modeling, they have proposed a particular type of neural networks containing nodes representing membership functions. In this fourth paper, the authors discuss input variable selection for the fuzzy modeling under multiple criteria with different importance. A target system with a strong nonlinearity is used for demonstrating the proposed method. Kasabov, et al. present, in the fifth paper, a method for extraction of fuzzy rules that have different level of abstraction depending on several modifiable thresholds. Explanation quality becomes better with higher threshold values. They apply the proposed method to the Iris benchmark classification problem and to a real world problem. J. Yen and W. Gillespie address interpretability issue of Takagi-Sugeno-Kang model, one of the most popular fuzzy mdoels, in the fifth paper. They propose a new approach of fuzzy modeling that ensures not only a high approximation of the input-output relationship in the data, but also good insights about the local behavior of the model. The proposed method is applied to fuzzy modeling of sinc function and Mackey-Glass chaotic time series data. The last paper discusses fuzzy rule extraction from numerical data for high-dimensional classification problems. H.Ishibuchi, et al. have been pioneering methods for classification of data using fuzzy rules and genetic algorithm. In this last paper, they introduced a new criterion, simplicity of each rule, together with the conventional ones, compactness of rule base and classification ability, for high-dimensional problem. The Iris data is used for demonstrating their new classification method. They applied it also to wine data and credit data. I hope that the readers will be encouraged to explore the frontier to establish a new paradigm in the field of knowledge representation and rule extraction.

Different optimization techniques are used for the training and fine-tuning of feed forward neural networks, for the estimation of STATCOM voltages and reactive powers. In the first part, the paper presents the voltage regulation in IEEE buses using the Static Compensator (STATIC) and discusses efficient ways to solve the power systems featuring STATCOM by load flow equations. The load flow equations are solved using iterative algorithms such as Newton-Raphson method. In the second part, the paper focuses on the use of estimation techniques based on Artificial Neural Networks as an alternative to the iterative methods. Different training algorithms have been used for training the weights of Artificial Neural Networks; these methods include Back-Propagation, Particle Swarm Optimization, Shuffled Frog Leap Algorithm, and Genetic Algorithm. A performance analysis of each of these methods is done on the IEEE bus data to examine the efficiency of each algorithm. The results show that SFLA outperforms other techniques in training of ANN, seconded by PSO.

In this paper, genetic programming (GP) combined with decision trees is used to evolve the structure and weights for artificial neural network (ANN). The learning rule of the decision tree technique is defined as a function of global information by employing a divide-and-conquer strategy. Learning rules with lower fitness values are replaced by new ones generated using GP techniques. The reciprocal connection between decision tree and GP emerges from the coordination of learning rules. Since there is no constraint on initial network structure, a more suitable network can be found for a given task. Further, the fitness values are improved by using a hybrid GP technique which is a combined technique of GP and back propagation. The proposed method is applied to a medical diagnostic system and experimental results demonstrate that effective learning rule evolves.