Application of Monte Carlo stochastic optimization (MOST) to deep learning

In this article we developed a method for optimizing the structure of a fuzzy artificial neural networks through genetic algorithms. This genetic algorithm is used by optimizing the number of weight connections in a neural network structure, by the evolution of those structures as individuals in a population. It is found that the optimization of the neural network provides higher confidence accuracy of the suggested solution in a case based diagnostic system. The computational cost of the optimized network also improved considerably high.

Linear combination rule in genetic algorithm for optimization of finite impulse response neural network to predict natural chaotic time series

The field of neural network training and optimization has seen significant advancements in recent years, with new techniques and algorithms being proposed to improve the efficiency and effectiveness of training. In this paper, we review several key optimization techniques and their impact on training neural networks, with a focus on long-term dependencies and the difficulties that can arise during training. We begin by discussing the challenges of learning long-term dependencies with gradient descent, as highlighted in the 1994 paper by Bengio et al. We then introduce Adam, a method for stochastic optimization proposed by Kingma and Ba in 2014. We also explore the difficulties of training recurrent neural networks, as discussed in the 2013 paper by Pascanu, Mikolov, and Bengio. We also review recent advances in optimization techniques such as Convergence of Adam and Beyond by Jianmin et al. (2017), Yogi by Dong et al. (2018), AdaBound by Zhang et al. (2018) and On the Variance of the Adaptive Learning Rate and Beyond by Liu et al. (2019). We will highlight the advantages and disadvantages of each technique and discuss their potential impact on the field. Overall, this paper provides a comprehensive overview of recent advancements in neural network optimization and their implications for training and performance. Keywords: Deep neural networks, optimization, Adam, gradient clipping, training, stabilization, overfitting, generalization, recurrent neural networks, non-convex loss landscapes.

Antenna modeling based on meta-heuristic intelligent algorithms and neural networks

Ellipsoidal basis function(EBF) can make the partition and limitary of input space. Compared with the Guassian function of radial basis function(RBF) neural network, the EBF can make the partition of input space more specific, which has the higher capability of pattern recognition. However, the neural network has a common problem of training the weight and threshold. The evolution of genetic algorithm(GA) can maximumly optimize the training time of neural network. In this paper, a new method based on GA-EBF neural network was proposed. The simulation experiment shows that the proposed method has a higher rate of fault diagnosis than that of RBF neural network. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>

Proper diagnosis of breast cancer is one of the main issues in a medical sector. The neural network could overcome the issue, but weak in determining parameter value, therefore it requires optimization. A genetic algorithm is one of the optimization methods, due to that matter, attribute parameter value of neural network will be optimized by using Genetic Algorithm to acquire the best predictor attributes. Genetic-based Neural Network algorithm has higher accuracy compared to the application of Neural Network algorithm alone. This notion is proven through the increasing rate of accuracy value for Neural Network algorithm which amounted to 96.57 % and 97.24 % for Genetic-Based Neural Network Algorithm as well as the decreasing rate of error classification of Neural Network algorithm model from 3.43 % rate to 2.86 % due to the application of Genetic-Based Neural Network Algorithm. Therefore, it can be concluded that the application of the optimization technique of Genetic Algorithm can enhance the accuracy value and decrease the error value of the Neural Network algorithm on breast cancer classification.

The Sixth International Symposium on Neural Networks (ISNN 2009)

The paper proposes a fast online learning method for neural network structures by using genetic algorithm (GA) and dynamic back propagation algorithm (BP) jointly. GA is used in the coarse tuning process which adjusts interconnection weights of the neural network. The dynamic back propagation algorithm is subsequently applied to achieve fine adjusting of the network weights. The fitness function, based on the squared error between the teaching signal and the network output value, is redefined at every time step and the proposed GA based algorithm solves a nonstationary function optimization task. At every time step the solution with the best fitness function is used for current representation of the neural network weights and biases. It is shown through the simulations and real time temperature control of drying oven that this learning algorithm has faster convergence ability and better performance on reducing mapping error in the online learning neural network structures. This leads to an improvement of the transient response of neuro adaptive systems. The proposed method has the potential to be applied to many practical areas such as system modeling and control, signal processing and pattern recognition.

The uncertainty of wind power as the main obstacle of its integration into the power grid can be addressed by an accurate and efficient wind power forecast. Among the various wind power forecasting methods, machine learning (ML) algorithms, are recognized as a powerful wind power forecasting tool, however, their performance is highly dependent on the proper tuning of their hyperparameters. Common hyperparameter tuning methods such as grid search or random search are time-consuming, computationally expensive, and unreliable for complex models such as deep learning neural networks. Therefore, there is an urgent need for automatic methods to discover optimal hyperparameters for higher accuracy and efficiency of prediction models. In this study, a novel investigation is contributed to the field of wind power forecasting by a comprehensive comparison of three advanced techniques – Scikit-opt, Optuna, and Hyperopt – for hyperparameter optimization of Convolutional Neural Network (CNN) and Long Short-Term Memory Network (LSTM) models, a facet that, to our knowledge, has not been systematically explored in existing literature. The impact of these optimization techniques on the accuracy and efficiency of the CNN and LSTM models are assessed by comparing the root mean square error (RMSE) of the predictions and the required time to tune the models. The results show that the Optuna algorithm, using a Tree-structured Parzen Estimator (TPE) search method and Expected Improvement (EI) acquisition function, has the best efficiency for both CNN and LSTM models. In terms of accuracy, it is demonstrated that while for the CNN model all the optimization methods achieve similar performances, the LSTM model optimized by the Hyperopt algorithm, based on the annealing search method, results in the highest accuracy. In addition, for the first time in this research, the impact of the random initialization features on the performance of the forecasting models with neural networks is investigated. The proposed structures for deep learning models were examined to determine the most robust structure with the minimal sensitivity to the randomness. What we have discovered from the comparison of advanced hyperparameter optimization methods can be used by researchers to tune the time series-based forecasting models.

Supervised Neural Network (NN) learning is a process in which input patterns and known targets are presented to a NN while it learns to recognize (classify, map, fit, etc.) them as desired. The learning is mathematically defined as an optimisation problem, i.e., an error function representing the differences between the desired and actual output, is being minimized (Bishop, 1995; Haykin, 1999). Because the most popular supervised learning techniques are gradient based (Backpropagation BP), they suffer from the so-called Local Minima Problem (Bishop, 1995). This has motivated the employment of Global Optimisation (GO) methods for the supervised NN learning. Stochastic and heuristic GO approaches including Evolutionary Algorithms (EA) demonstrated promising performance over the last decades (Smagt, 1994; Sexton et al., 1998; Jordanov & Georgieva, 2007; etc.). EA appeared more powerful than BP and its modifications (Sexton et al., 1998; Alba & Chicone 2004), but hybrid methods that combine the advantages of one or more GO techniques and local searches were proven to be even better (Yao, 1999; Rocha et al., 2003; Alba & Chicano, 2004; Ludemir et al., 2006). Hybrid methods were promoted over local searches and simple population based techniques in Alba & Chicone (2004). The authors compared five methods: two BP implementations (gradient descent and Levenberg-Marquardt), Genetic Algorithms (GA), and two hybrid methods, combining GA with different local methods. The methods were used for NN learning applied to problems arising in medicine. Ludemir et al. (2006) optimized simultaneously NN weights and topology with a hybrid method combining Simulated Annealing (SA), Tabu Search (TS) and BP. A set of new solutions was generated on each iteration by TS rules, but the best solution was only accepted according to the probability distribution as in conventional SA. Meanwhile, the topology of the NN was also optimized and the best solution was kept. Finally, BP was used to train the best NN topology found in the previous stages. The new methodology compared favorably with SA and TS on four classification and one prediction problems. Plaginakos et al. (2001) performed several experiments to evaluate various training methods – six Differential Evolution (DE) implementations (with different mutation operators), BP, BPD (BP with deflection), SA, hybridization of BP and SA (BPSA), and GA. They reported O pe n A cc es s D at ab as e w w w .in te ch w eb .o rg

Calculus equation is an important tool for mathematical research and plays an important role in most natural science research. Since the beginning of the eighteenth century, people have gradually used differential and integral equations to solve physical problems. In general, several different aspects of differential equations in the field of mathematics are concerned and studied by most scholars. However, this paper studies and establishes the optimal model for numerical solution of differential equations through deep learning and genetic algorithm. In this paper, the solution of ordinary differential equations is solved through the use of polynomial function space, while the linear combination of simple function x and its product nx can obtain multinomial function space. The space function form of polynomial is very simple, and the operation ability is very strong. Almost all functions can be approximated, and the function space can be transformed by a simple function. Through data simulation test results, it can be found that the oscillation of neural network output is stronger and stronger with the increase in depth, that is to say, the deeper depth endows the neural network with stronger oscillation properties, so for the oscillation function, the depth neural network fitting effect is better than the shallow neural network. Therefore, by combining deep learning and genetic algorithm, this paper studies and establishes the optimal model for numerical solution of differential equations, and finds that the deep neural network can largely complete data simulation.

Rule extraction from data is one of the key technologies for solving the bottlenecks in artificial intelligence. Artificial neural networks are well suited for representing any knowledge in given data. Extraction of logical/fuzzy rules from the trained artificial neural network is of great importance to researchers in the fields of artificial intelligence and soft computing. Fuzzy rule sets are capable of approximating any nonlinear mapping relationships. Extraction of rules from data has been discussed in terms of fuzzy modeling, fuzzy clustering, and classification with fuzzy rule sets. This special issue entitled"Rule Extraction from Data" is aimed at providing the readers with good insights into the advanced studies in the field of rule extraction from data using neural networks/fuzzy rule sets. I invited seven research papers best suited for the theme of this special issue. All the papers were reviewed rigorously by two reviewers each. The first paper proposes an interesting rule extraction method from data using neural networks. Ishikawa presents a combination of learning with an immediate critic and a structural learning with forgetting. This method is capable of generating skeletal networks for logical rule extraction from data with correct and wrong answers. The proposed method is applied to rule extraction from lense data. The second paper presents a new methodology for logical rule extraction based on transformation of MLP (multilayered perceptron) to a logical network. Duck et al. applied their C-MLP2LN to the Iris benchmark classification problem as well as real-world medical data with very good results. In the third paper, Geczy and Usui propose fuzzy rule extraction from trained artificial neural networks. The proposed algorithm is implied from their theoretical study, not from heuristics. Their study enables to initially consider derivation of crisp rules from trained artificial neural network, and in case of conflict, application of fuzzy rules. The proposed algorithm is experimentally demonstrated with the Iris benchmark classification problem. The fourth paper presents a new framework for fuzzy modeling using genetic algorithm. The authors have broken new ground of fuzzy rule extraction from neural networks. For the fuzzy modeling, they have proposed a particular type of neural networks containing nodes representing membership functions. In this fourth paper, the authors discuss input variable selection for the fuzzy modeling under multiple criteria with different importance. A target system with a strong nonlinearity is used for demonstrating the proposed method. Kasabov, et al. present, in the fifth paper, a method for extraction of fuzzy rules that have different level of abstraction depending on several modifiable thresholds. Explanation quality becomes better with higher threshold values. They apply the proposed method to the Iris benchmark classification problem and to a real world problem. J. Yen and W. Gillespie address interpretability issue of Takagi-Sugeno-Kang model, one of the most popular fuzzy mdoels, in the fifth paper. They propose a new approach of fuzzy modeling that ensures not only a high approximation of the input-output relationship in the data, but also good insights about the local behavior of the model. The proposed method is applied to fuzzy modeling of sinc function and Mackey-Glass chaotic time series data. The last paper discusses fuzzy rule extraction from numerical data for high-dimensional classification problems. H.Ishibuchi, et al. have been pioneering methods for classification of data using fuzzy rules and genetic algorithm. In this last paper, they introduced a new criterion, simplicity of each rule, together with the conventional ones, compactness of rule base and classification ability, for high-dimensional problem. The Iris data is used for demonstrating their new classification method. They applied it also to wine data and credit data. I hope that the readers will be encouraged to explore the frontier to establish a new paradigm in the field of knowledge representation and rule extraction.

Optimization and design of an aircraft’s morphing wing-tip demonstrator for drag reduction at low speed, Part I – Aerodynamic optimization using genetic, bee colony and gradient descent algorithms

Neural networks with ReLU powers need less depth

A new classification framework for a Clinical Decision Support System, utilizing a Genetic algorithm and an Artificial Flora Optimized Neural Network is presented in this paper. GAFON is an artificial neural network whose topology is optimized with Genetic Algorithm and the learnable parameters are optimized with Artificial Flora Optimization algorithm. Drop out technique is used in the topology optimization phase and weight regularization is used in the parameter optimization phase. The proposed method minimizes the co-adaptation problem, reduces over-fitting of training data and improves the generalization of a feed forward neural network. The classification framework developed has been tested for classifying both multi class and binary class clinical datasets. The proposed method attained accuracy values of 86.82% for Hepatitis C Virus (HCV) for Egyptian patients, 84.91% for Vertebral Column 95.65% for Statlog Heart Disease (SHD), SHD and 93.79% for Early Stage Diabetes Risk Prediction (ESDRP), all datasets obtained from UCI repository