Knowledge Extraction from Artificial Neural Networks for Rainfall-Runoff Model Combination Systems

Artificial neural networks have proven to be effective in a wide range of fields, providing solutions to various problems. Training artificial neural networks using evolutionary algorithms is known as neuroevolution. The idea of finding not only the optimal weights and biases of a neural network but also its architecture has drawn the attention of many researchers. In this paper, we use different biologically inspired optimization algorithms to train multilayer perceptron neural networks for generating regression models. Specifically, our contribution involves analyzing and finding a strategy for combining several algorithms into a hybrid ensemble optimizer, which we apply for the optimization of a fully connected neural network. The goal is to obtain good regression models for studying and making predictions for the process of free radical polymerization of methyl methacrylate (MMA). In the first step, we use a search procedure to find the best parameter values for seven biologically inspired optimization algorithms. In the second step, we use a subset of the best-performing algorithms and improve the search capability by combining the chosen algorithms into an ensemble of optimizers. We propose three ensemble strategies that do not involve changes in the logic of optimization algorithms: hybrid cascade, hybrid single elite solution, and hybrid multiple elite solutions. The proposed strategies inherit the advantages of each individual optimizer and have faster convergence at a computational effort very similar to an individual optimizer. Our experimental results show that the hybrid multiple elite strategy ultimately produces neural networks which constitute the most dependable regression models for the aforementioned process.

A scheme to predict transient stability status following a disturbance is presented. The scheme is activated upon the tripping of a line or bus and operates as follows: Samples of frequency deviations of all generator buses are obtained using a sampling frequency of 32 samples per cycle. For each generator bus, the maximum frequency deviation within the first two samples is obtained. A vector is then constructed from the obtained maximum frequency deviations. The Euclidean norm of the vector is calculated and then fed as input to a trained multilayer perceptron neural network which predicts the stability status of the system. The scheme was tested using the New England test system. The prediction accuracy was found to be 99.49%.

BACKGROUND: Currently, artificial intelligence in the form of artificial neural networks is being actively implemented in a number of areas of our lives, including medicine. In particular, in otorhinolaryngology, artificial neural networks are used to analyze images obtained during endoscopic examinations of patients (e.g., videolaryngoscopy) [1–3]. The interpretation of laryngoscopic images often presents significant difficulties for practicing physicians, which reduces the frequency of detection of precancerous laryngeal diseases and contributes to the increase in the number of patients with stage III–IV laryngeal cancer [4, 5]. This underscores the significance of prompt performance and accurate interpretation of the findings of endoscopic examinations of patients with laryngeal disorders. Artificial neural networks can be employed to analyze the results of videolaryngoscopy, furnishing the physician with supplementary information that can enhance diagnostic accuracy and diminish the probability of error [6, 7]. AIM: The study aims to develop and train an artificial neural network for recognizing characteristic features of laryngeal neoplasms and variants of laryngeal normality. MATERIALS AND METHODS: The study was conducted under the grant of the Moscow Center for Innovative Technologies in Healthcare (grant No. 2112-1/22) entitled “Using Neural Networks (Artificial Intelligence Algorithms) for Control and Improving the Quality of Diagnosis and Treatment of Diseases of Laryngeal and Ear Structures through Digital Technologies”.The following methods were used during the course of the study: data collection for the creation of a photobank (dataset) of medical images obtained during videolaryngoscopy; data partitioning for the formation of datasets for individual nosologies and groups of diseases; the method of consilium; analysis of the accuracy of recognition and classification of digital endoscopic images; and training of classification neural networks. Consequently, a dataset comprising 1,471 laryngeal images in digital formats (JPEG, BMP) was assembled, labelled, and uploaded for the purpose of training the artificial neural network. Of the total number of images, 410 were classified as pertaining to laryngeal formation, while 1061 were classified as variants of normality. Subsequently, the neural network was trained and tested to identify the signs of normal and laryngeal masses. RESULTS: The results of the testing of the artificial neural network indicated the formation of an inaccuracy matrix, the calculation of the value of recognition accuracy, the calculation of the quality indicators of the model performance, and the construction of the ROC curve. The developed and trained artificial neural network demonstrated an accuracy of 86% in recognizing the signs of laryngeal masses and norms. CONCLUSIONS: This study demonstrates that a trained artificial neural network can successfully distinguish between signs of normal and laryngeal masses in endoscopic photographs. With further training of the neural network and achievement of high accuracy, this technology can be used in clinical practice as an assistant in the interpretation of laryngoscopic images and early diagnosis of laryngeal masses. It can also be employed to control and improve the quality of diagnosis and treatment of diseases of the throat, nose, and ears by primary care physicians.

Artificial neural networks (ANN) have been widely used in the field of data classification. Normally, training of neural network is applied with the traditional back propagation technique. As, this approach has various drawbacks, training of neural network is done with Particle Swarm Optimization (PSO). PSO has been widely used to solve the diverse kind of optimization problems. Population initialization performs a significant role in meta-heuristic algorithms. This paper describes a new initialization population approach Log Logistic termed as PSOLL-NN to create the initialization of the swarm. The proposed algorithm has been tested for weight optimization of feed forward neural network; and compared with back propagation Algorithm (BPA), standard PSO (PSO-NN), PSO initialized with Halton Sequence (PSOH-NN), Torus sequence (PSOT-NN) and Sobol sequence (PSOS-NN). The experimental results show that the proposed technique performed exceptionally better than the other traditional techniques. Moreover, the outcome of our work presents a foresight that how the proposed initialization technique can be used as an efficient alternative to standard training approaches for the data classification problems.

Looking at the artificial neural networks’ literature, most of the studies started with feedforward artificial neural networks and the training of many feedforward artificial neural networks models are performed with derivative-based algorithms such as levenberg–marquardt and back-propagation learning algorithms in the first studies. In recent years, although many new heuristic algorithms have been proposed for different aims these heuristic algorithms are also frequently used in the training process of many different artificial neural network models. Pi-sigma artificial neural networks have different importance than many artificial neural network models with its higher-order network structure and superior forecasting performance. In this study, the training of Pi-Sigma artificial neural networks is performed by differential evolution algorithm uses DE/rand/1 mutation strategy. The performance of the proposed method is evaluated by two data sets and seen that the proposed method has a very effective performance compared with many artificial neural network models.

Joint A-SNN: Joint training of artificial and spiking neural networks via self-Distillation and weight factorization

This study presents establishment of multiple input prediction model for automotive coil spring fatigue life estimation to shorten automotive suspension design process. Automotive suspension design is a lengthy work where any changes of the design lead to repetition of the entire process. It was hypothesised that the established model could be used to predict the spring design fatigue life without using any strain measurements. To initiate this model establishment, five sets of strain and acceleration measurement across different road conditions were collected and used for validations. To include spring stiffness as a parameter, a quarter car model was generated to obtain the force time histories from spring and vertical vibration of vehicle mass. In addition, artificial road profiles of road classes “A” to “D” were also generated for the quarter car simulation. Through adjusting the spring stiffness in the quarter car model, the spring and vehicle responses were varied. The simulated force time histories were used to predict springs’ fatigue life while acceleration time histories were used to calculate ISO 2631 ride-related vertical vibration. Subsequently, multiple linear regression approach was applied to determine the relationship between vehicle body frequency, ISO 2631 ride-related vertical vibration and spring fatigue life. The obtained regression had shown significance to the spring fatigue life with coefficient of determination of 0.8320. Reciprocally, multiple linear regression models were also used to predict the ISO 2631 ride-related vertical vibration with a coefficient of determination at 0.8810 and mean squared error values below 0.3430. To optimise the prediction results, artificial neural network was trained for the fatigue and vibration prediction purposes. The architectures of the artificial neural network were designed in terms of number of neurons and hidden layers to achieve a higher coefficient of determination of 0.9926 and lower mean squared error of 0.0824. For vibration prediction, the vehicle body frequency and spring fatigue life has shown a significant coefficient of determination to the ISO 2631 weighted vertical vibration, reaching 0.9579 with mean squared error of 0.0004. Based on the experimental strain and acceleration results, the predicted fatigue lives of multiple linear regression models were correlated well with the experimental results with coefficient of determination value of 0.9275. Meanwhile, the maximum difference of vibration prediction to experimental value using multiple linear regression models was only 18%. For artificial neural network predictions, the fatigue lives were mostly distributed within 1:2 or 2:1 life correlation and vibration prediction results were within 12%. For a good prediction, the target correlation value was above 0.80 to demonstrate a good fitted curve and the difference below 20%. The trained artificial neural network has shown outstanding capability in fatigue life or ride-related vertical vibration predictions. In this research, the main novelty was the trained artificial neural network for spring fatigue life or ride-related vertical vibration predictions which serve to reduce some procedures of automotive suspension design. The outcome of this study can be used to provide a new knowledge towards the field of fatigue research as well as vehicle ride dynamics. This research contributes to automotive industries especially in suspension spring design where the analysis of fatigue and ride-related vibration are provided.

Optimum design of aerospace structural components using neural networks

This paper presents an improved artificial neural network (ANN) training using response surface methodology (RSM) optimization for membrane flux prediction. The improved ANN utilizes the design of experiment (DoE) technique to determine the neural network parameters. The technique has the advantage of training performance, with a reduced training time and number of repetitions in achieving good model prediction for the permeate flux of palm oil mill effluent. The conventional training process is performed by the trial-and-error method, which is time consuming. In this work, Levenberg–Marquardt (lm) and gradient descent with momentum (gdm) training functions are used, the feed-forward neural network (FFNN) structure is applied to predict the permeate flux, and airflow and transmembrane pressure are the input variables. The network parameters include the number of neurons, the learning rate, the momentum, the epoch, and the training functions. To realize the effectiveness of the DoE strategy, central composite design is incorporated into neural network methodology to achieve both good model accuracy and improved training performance. The simulation results show an improvement of more than 50% of training performance, with less repetition of the training process for the RSM-based FFNN (FFNN-RSM) compared with the conventional-based FFNN (FFNN-lm and FFNN-gdm). In addition, a good accuracy of the models is achieved, with a smaller generalization error.

Prediction of student performance is one of the most important subjects of educational data mining. Artificial neural networks are seen to be an effective tool in predicting student performance in e-learning environments. In the studies carried out with artificial neural networks, performance predictions based on student scores are generally made, but students’ use of learning management system is not focused. In this study, performances of 3518 university students, who studying and actively participating in a learning management system, were tried to be predicted by artificial neural networks in terms of gender, content score, time spent on the content, number of entries to content, homework score, number of attendance to live sessions, total time spent in live sessions, number of attendance to archived courses and total time spent in archived courses variables. Since it is difficult to interpret how much input variables in artificial neural networks contribute to predicting output variables, these networks are called black boxes. Also, in this study the amount of contribution of input variables on the prediction of output variable was also examined. The artificial neural network created as a result of the study makes a prediction with an accuracy of 80.47%. Finally, it was found that the variables of number of attendance to the live classes, the number of attendance to archived courses and the time spent in the content contributed most to the prediction of the output variable.

This paper presents a novel technique for the supervised training of feed-forward artificial neural networks (ANN) using the Harmony Search (HS) algorithm. HS is a stochastic meta-heuristic that is inspired from the improvisation process of musicians. Unlike Backpropagation, HS is non-trajectory driven. By modifying an existing improved version of HS & adopting a suitable ANN data representation, we propose a training technique where two of HS probabilistic parameters are determined dynamically based on the best-to-worst (BtW) harmony ratio in the current harmony memory instead of the improvisation count. This would be more suitable for ANN training since parameters and termination would depend on the quality of the attained solution. We have empirically tested and verified our technique by training an ANN with a benchmarking problem. In terms of overall training time and recognition, our results have revealed that our method is superior to both the original improved HS and standard Backpropagation.

<abstract><p>The training of artificial neural networks (ANNs) with rectified linear unit (ReLU) activation via gradient descent (GD) type optimization schemes is nowadays a common industrially relevant procedure. GD type optimization schemes can be regarded as temporal discretization methods for the gradient flow (GF) differential equations associated to the considered optimization problem and, in view of this, it seems to be a natural direction of research to <italic>first aim to develop a mathematical convergence theory for time-continuous GF differential equations</italic> and, thereafter, to aim to extend such a time-continuous convergence theory to implementable time-discrete GD type optimization methods. In this article we establish two basic results for GF differential equations in the training of fully-connected feedforward ANNs with one hidden layer and ReLU activation. In the first main result of this article we establish in the training of such ANNs under the assumption that the probability distribution of the input data of the considered supervised learning problem is absolutely continuous with a bounded density function that every GF differential equation admits for every initial value a solution which is also unique among a suitable class of solutions. In the second main result of this article we prove in the training of such ANNs under the assumption that the target function and the density function of the probability distribution of the input data are piecewise polynomial that every non-divergent GF trajectory converges with an appropriate rate of convergence to a critical point and that the risk of the non-divergent GF trajectory converges with rate 1 to the risk of the critical point. We establish this result by proving that the considered risk function is <italic>semialgebraic</italic> and, consequently, satisfies the <italic>Kurdyka-Łojasiewicz inequality</italic>, which allows us to show convergence of every non-divergent GF trajectory.</p></abstract>

Robotic airships have numerous low-speed and low-altitude application potentials. Mission path following is one such application, which, however, presents an autonomy challenge. In this paper, a yawing controller, which is based on artificial neural network (ANN) and human operator skills, is proposed for mission path following of robotic airships. First, the path-following errors based on the operator’s point of view are discussed. Then, a data acquisition system is designed to collect the flight data under manual control, and the data are then processed and used for offline training and validation of a multilayer feed-forward neural network. Finally, the trained neural network is reconstructed in the flight control system for yawing control, and the experimental results confirm the effectiveness of this method. It is also shown that the ANN controller is robust even with wind disturbance.

The paper presents laboratory implementation of a photovoltaic artificial neural network (ANN) based maximum power tracking controller. The control purpose is to track the maximum available solar power in a photovoltaic array interfaced to an electric utility grid via a line-commutated inverter. The inverse dynamic characteristics of this scheme is identified by off-line training of a multilayer perceptron type neural network. The ANN output is used as the control signal to vary the line-commutated inverter firing angle, hence track the available maximum solar power. The weights of the ANN is updated by an online training algorithm which utilizes the online power mismatch error. This ensures online maximum solar power tracking. >

This paper focuses on one half of the knowledge acquisition problem for fuzzy systems, namely the acquisition of a fuzzy rule base from a set of input/output data, and in particular on the extraction of a set of fuzzy rules from a trained neural network. Some limitations with previously reported work in this area are first identified. Two simple rule extraction techniques are then described and tested on a well known classification problem. The performance of the resultant rule bases compares more favourably than those reported using alternative techniques.