Training of Artificial Neural Network Using PSO With Novel Initialization Technique

BACKGROUND: Currently, artificial intelligence in the form of artificial neural networks is being actively implemented in a number of areas of our lives, including medicine. In particular, in otorhinolaryngology, artificial neural networks are used to analyze images obtained during endoscopic examinations of patients (e.g., videolaryngoscopy) [1–3]. The interpretation of laryngoscopic images often presents significant difficulties for practicing physicians, which reduces the frequency of detection of precancerous laryngeal diseases and contributes to the increase in the number of patients with stage III–IV laryngeal cancer [4, 5]. This underscores the significance of prompt performance and accurate interpretation of the findings of endoscopic examinations of patients with laryngeal disorders. Artificial neural networks can be employed to analyze the results of videolaryngoscopy, furnishing the physician with supplementary information that can enhance diagnostic accuracy and diminish the probability of error [6, 7]. AIM: The study aims to develop and train an artificial neural network for recognizing characteristic features of laryngeal neoplasms and variants of laryngeal normality. MATERIALS AND METHODS: The study was conducted under the grant of the Moscow Center for Innovative Technologies in Healthcare (grant No. 2112-1/22) entitled “Using Neural Networks (Artificial Intelligence Algorithms) for Control and Improving the Quality of Diagnosis and Treatment of Diseases of Laryngeal and Ear Structures through Digital Technologies”.The following methods were used during the course of the study: data collection for the creation of a photobank (dataset) of medical images obtained during videolaryngoscopy; data partitioning for the formation of datasets for individual nosologies and groups of diseases; the method of consilium; analysis of the accuracy of recognition and classification of digital endoscopic images; and training of classification neural networks. Consequently, a dataset comprising 1,471 laryngeal images in digital formats (JPEG, BMP) was assembled, labelled, and uploaded for the purpose of training the artificial neural network. Of the total number of images, 410 were classified as pertaining to laryngeal formation, while 1061 were classified as variants of normality. Subsequently, the neural network was trained and tested to identify the signs of normal and laryngeal masses. RESULTS: The results of the testing of the artificial neural network indicated the formation of an inaccuracy matrix, the calculation of the value of recognition accuracy, the calculation of the quality indicators of the model performance, and the construction of the ROC curve. The developed and trained artificial neural network demonstrated an accuracy of 86% in recognizing the signs of laryngeal masses and norms. CONCLUSIONS: This study demonstrates that a trained artificial neural network can successfully distinguish between signs of normal and laryngeal masses in endoscopic photographs. With further training of the neural network and achievement of high accuracy, this technology can be used in clinical practice as an assistant in the interpretation of laryngoscopic images and early diagnosis of laryngeal masses. It can also be employed to control and improve the quality of diagnosis and treatment of diseases of the throat, nose, and ears by primary care physicians.

<abstract><p>The training of artificial neural networks (ANNs) with rectified linear unit (ReLU) activation via gradient descent (GD) type optimization schemes is nowadays a common industrially relevant procedure. GD type optimization schemes can be regarded as temporal discretization methods for the gradient flow (GF) differential equations associated to the considered optimization problem and, in view of this, it seems to be a natural direction of research to <italic>first aim to develop a mathematical convergence theory for time-continuous GF differential equations</italic> and, thereafter, to aim to extend such a time-continuous convergence theory to implementable time-discrete GD type optimization methods. In this article we establish two basic results for GF differential equations in the training of fully-connected feedforward ANNs with one hidden layer and ReLU activation. In the first main result of this article we establish in the training of such ANNs under the assumption that the probability distribution of the input data of the considered supervised learning problem is absolutely continuous with a bounded density function that every GF differential equation admits for every initial value a solution which is also unique among a suitable class of solutions. In the second main result of this article we prove in the training of such ANNs under the assumption that the target function and the density function of the probability distribution of the input data are piecewise polynomial that every non-divergent GF trajectory converges with an appropriate rate of convergence to a critical point and that the risk of the non-divergent GF trajectory converges with rate 1 to the risk of the critical point. We establish this result by proving that the considered risk function is <italic>semialgebraic</italic> and, consequently, satisfies the <italic>Kurdyka-Łojasiewicz inequality</italic>, which allows us to show convergence of every non-divergent GF trajectory.</p></abstract>

This paper presents a novel technique for the supervised training of feed-forward artificial neural networks (ANN) using the Harmony Search (HS) algorithm. HS is a stochastic meta-heuristic that is inspired from the improvisation process of musicians. Unlike Backpropagation, HS is non-trajectory driven. By modifying an existing improved version of HS & adopting a suitable ANN data representation, we propose a training technique where two of HS probabilistic parameters are determined dynamically based on the best-to-worst (BtW) harmony ratio in the current harmony memory instead of the improvisation count. This would be more suitable for ANN training since parameters and termination would depend on the quality of the attained solution. We have empirically tested and verified our technique by training an ANN with a benchmarking problem. In terms of overall training time and recognition, our results have revealed that our method is superior to both the original improved HS and standard Backpropagation.

Particle swarm optimisation (PSO) was developed by Kennedy and Eberhart in 1995 (Kennedy & Eberhart, 1995) inspired by the collective behaviour of natural birds or fish. PSO is a stochastic optimisation technique that uses a behaviour of population composed by many search points called particle. In spite of easy implementation in computer algorithms, it is well known as a powerful numerical optimizer. In the typical PSO algorithms, a set of particles searches the optimal solution in the problem space efficiently, by sharing the common attractor called global best. There are many modified versions of PSO by improving convergence property to a certain problem. While, a standard PSO is defined by Bratton and Kennedy (Bratton & Kennedy, 2007) to give a real standard for PSO studies. PSO seems as one of the evolutionary computations (ECs), and it has been shown that PSO is comparable to a genetic algorithm (Angeline, 1998). Thus, a lot of studies have demonstrated the effectiveness of PSO family in optimizing various continuous and discrete optimization problems. And a plenty of applications of PSO, such as the neural network training, PID controller tuning, electric system optimisation have been studied and achieved well results (Kennedy, 1997). However, PSO is often failed in searching the global optimal solution in the case of the objective function has a large number of dimensions. The reason of this phenomenon is not only existence of the local optimal solutions, the velocities of the particles sometimes lapsed into the degeneracy, so that the successive range is restricted in the sub-plain of the whole search hyper-plain. The sub-plane that is defined by finite number of particle velocities is a partial space in the whole search space. The issue of local optima in PSO has been studied and proposed several modifications on the basic particle driven equation (Parsopoulos et al., 2001; Hendtlass, 2005; Liang et al., 2006). There used a kind of adaptation technique or randomized method (e.g. mutation in evolutionary computations) to keep particles velocities or to accelerate them. Although such improvements work well and have ability to avoid fall in the local optima, the problem of early convergence by the degeneracy of some dimensions is still remaining, even if there are no local optima. Hence the PSO algorithm does not always work well for the high-dimensional function.

Relevance. Modern artificial neural network models require significant energy and other resources for training and operation. Training generative models involves vast amounts of data. At the same time, these models face challenges related to the trustworthiness of the information they generate. An alternative to current paradigms of building and training neural networks is the development of energy-based models, which could potentially overcome these shortcomings and bring information processing closer to biologically and physically grounded processes. However, existing energy-based models differ little from classical models in terms of their limitations and drawbacks. Therefore, developing new approaches to modeling energy-based information processing in neural networks is highly relevant. The object of research is the process of information processing in artificial neural networks. The subject of the research is the mathematical models for the construction and training of artificial neural networks. The purpose of this paper is to develop and experimentally validate a theoretical framework that postulates the energetic nature of information and its role in the self-organization and evolution of complex information systems. Research Results. A fundamental theory is proposed, describing information as a structure of perceived external energy parameters that govern the processes of forming the internal energetic structure of a system—its model of the external world. This theory encompasses concepts of energy landscapes, principles of energy-based structural and parametric reduction, and a critical analysis of existing computational paradigms. Experimental studies on the construction and training of the developed energy-based model confirm its high generalization ability in one-pass training without using the backpropagation algorithm on ultra-small training datasets.

One of the most used artificial intelligence techniques for maximum power point tracking is artificial neural networks. In order to achieve successful results in maximum power point tracking, the training process of artificial neural networks is important. Metaheuristic algorithms are used extensively in the literature for neural network training. An important group of metaheuristic algorithms is swarm-intelligent-based optimization algorithms. In this study, feed-forward neural network training is carried out for maximum power point tracking by using 13 swarm-intelligent-based optimization algorithms. These algorithms are artificial bee colony, butterfly optimization, cuckoo search, chicken swarm optimization, dragonfly algorithm, firefly algorithm, grasshopper optimization algorithm, krill herd algorithm, particle swarm optimization, salp swarm algorithm, selfish herd optimizer, tunicate swarm algorithm, and tuna swarm optimization. Mean squared error is used as the error metric, and the performances of the algorithms in different network structures are evaluated. Considering the results, a success ranking score is obtained for each algorithm. The three most successful algorithms in both training and testing processes are the firefly algorithm, selfish herd optimizer, and grasshopper optimization algorithm, respectively. The training error values obtained with these algorithms are 4.5 × 10−4, 1.6 × 10−3, and 2.3 × 10−3, respectively. The test error values are 4.6 × 10−4, 1.6 × 10−3, and 2.4 × 10−3, respectively. With these algorithms, effective results have been achieved in a low number of evaluations. In addition to these three algorithms, other algorithms have also achieved mostly acceptable results. This shows that the related algorithms are generally successful ANFIS training algorithms for maximum power point tracking.

Artificial neural networks (ANNs) are used to solve many problems, such as modeling, identification, prediction, and classification. The success of ANN is directly related to the training process. Meta-heuristic algorithms are used extensively for ANN training. Within the scope of this study, a feed-forward artificial neural network (FFNN) is trained using the marine predators algorithm (MPA), one of the current meta-heuristic algorithms. Namely, this study is aimed to evaluate the performance of MPA in ANN training in detail. Identification/modeling of nonlinear systems is chosen as the problem. Six nonlinear systems are used in the applications. Some of them are static, and some are dynamic. Mean squared error (MSE) is utilized as the error metric. Effective training and testing results were obtained using MPA. The best mean error values obtained for six nonlinear systems are 2.3 × 10−4, 1.8 × 10−3, 1.0 × 10−4, 1.0 × 10−4, 1.2 × 10−5, and 2.5 × 10−4. The performance of MPA is compared with 16 meta-heuristic algorithms. The results have shown that the performance of MPA is better than other algorithms in ANN training for the identification of nonlinear systems.

In this work a modification was made on the algorithm of Artificial Neural Networks (NN) Training of the Multilayer Perceptron type (MLP) based on multi-objective optimization (MOBJ), to increase its computational efficiency. Usually, the number of efficient solutions to be generated is a parameter that must be provided by the user. In this work, this number is automatically determined by an algorithm, through the usage of golden section, being generally less when specified, showing a sensible reduction in the processing time and keeping the high generalization capability of the obtained solution from the original method.

Strategically injected noise can speed up convergence during Neural Network training using backpropagation algorithm. Noise injection during Neural Network training have been proven empirically to improve convergence and generalizability. In this work, a new methodology proven to be efficient for speeding up learning convergence using weight noise in Single Layer Feed-forward Network (SLFN) architecture is presented. We present efficient and effective methods in which local minimum entrapment can be avoided. Our proposed controlled introduction of noise is based on 4 proven analytical and experimental methods. We show that criteria-based mini-batch noise injection to the weights during training often outperforms the noiseless weights as well as fixed noise introduction as seen in literature both in network generalization and convergence speed. The effectiveness of this methodology has been empirically shown as well as it achieving on an average 15%–25% improvement in convergence speed when compared to fixed and noiseless networks. The proposed method is evaluated on the MNIST dataset and other datasets from UCI repository. The comparative analysis confirms that the proposed method achieves superior performance regarding convergence speed.

In this paper, we present a new key generation model for image hashing using neural network, which does not embed any data into the content but is able to extract meaningful data from target image. This model trains artificial neural network to assign predefined code and uses this trained artificial neural network weight and the coordinates of the selected feature sub blocks of target image as keys to extract the predefined code. In this model, the observed output signal from the trained neural network is used as image hash value which distinguishes the target image from other images. The proposed method contributes to secure image hashing for content identification without damaging or losing any detailed data of visual images. The proposed method realizes an application for image authentication, image similarity comparison, verification of image integrity and copyright protection of multimedia contents.

Levenberg-Marquardt back-propagation (LMBP) algorithm is one of the most widely used alternative algorithms for training of artificial neural networks (ANN). In some cases, learning in LM can become a challenging machine-learning task owing to poor selection of weights and biases. In this manner, the network may become unstable and cause the search process to get stuck in local minima (i.e. resulting into network stagnancy). Recently, meta-heuristic algorithms have proven themselves to alleviate such problem and improve the convergence slow convergence in ANN. In this paper, Wolf Search (WS) that simulates the hunting behavior of wolves in their natural habitat is implemented with LMBP algorithm to optimize the weight selection in ANN. The performance of the proposed Wolf Search with Levenberg-Marquardt (WSLM) is compared with back-propagation and other stochastic LM algorithms by means of simulations on benchmark datasets. The simulation results show that the proposed WSLM performs superior to BPNN, ABC-BP, Bat-BP, and ABC-LM algorithms in terms of convergence speed and local minima avoidance.

ABSTRACTA back‐propagation neural network was applied to predicting the KIC values using tensile material data and investigating the effects of crack plane orientation and temperature. The 595 KIC data of structural steels were used for training and testing the neural network model. In the trained neural network model, yield stress has relatively the most effect on KIC value among tensile material properties and KIC value was more sensitive to KIC test temperature than to crack plane orientation valid in the range of material data covered in this study. The performance of the trained artificial neural network (ANN) was evaluated by comparing output of the ANN with results of a conventional least squares fit to an assumed shape. The conventional linear or nonlinear least squares fitting methods gave very poor fitting results but the results predicted by the trained neural network were considerably satisfactory. This study shows that the neural network can be a good tool to predict KIC values according to the variation of the temperature and the crack plane orientation using tensile test results.

Looking at the artificial neural networks’ literature, most of the studies started with feedforward artificial neural networks and the training of many feedforward artificial neural networks models are performed with derivative-based algorithms such as levenberg–marquardt and back-propagation learning algorithms in the first studies. In recent years, although many new heuristic algorithms have been proposed for different aims these heuristic algorithms are also frequently used in the training process of many different artificial neural network models. Pi-sigma artificial neural networks have different importance than many artificial neural network models with its higher-order network structure and superior forecasting performance. In this study, the training of Pi-Sigma artificial neural networks is performed by differential evolution algorithm uses DE/rand/1 mutation strategy. The performance of the proposed method is evaluated by two data sets and seen that the proposed method has a very effective performance compared with many artificial neural network models.

This study presents establishment of multiple input prediction model for automotive coil spring fatigue life estimation to shorten automotive suspension design process. Automotive suspension design is a lengthy work where any changes of the design lead to repetition of the entire process. It was hypothesised that the established model could be used to predict the spring design fatigue life without using any strain measurements. To initiate this model establishment, five sets of strain and acceleration measurement across different road conditions were collected and used for validations. To include spring stiffness as a parameter, a quarter car model was generated to obtain the force time histories from spring and vertical vibration of vehicle mass. In addition, artificial road profiles of road classes “A” to “D” were also generated for the quarter car simulation. Through adjusting the spring stiffness in the quarter car model, the spring and vehicle responses were varied. The simulated force time histories were used to predict springs’ fatigue life while acceleration time histories were used to calculate ISO 2631 ride-related vertical vibration. Subsequently, multiple linear regression approach was applied to determine the relationship between vehicle body frequency, ISO 2631 ride-related vertical vibration and spring fatigue life. The obtained regression had shown significance to the spring fatigue life with coefficient of determination of 0.8320. Reciprocally, multiple linear regression models were also used to predict the ISO 2631 ride-related vertical vibration with a coefficient of determination at 0.8810 and mean squared error values below 0.3430. To optimise the prediction results, artificial neural network was trained for the fatigue and vibration prediction purposes. The architectures of the artificial neural network were designed in terms of number of neurons and hidden layers to achieve a higher coefficient of determination of 0.9926 and lower mean squared error of 0.0824. For vibration prediction, the vehicle body frequency and spring fatigue life has shown a significant coefficient of determination to the ISO 2631 weighted vertical vibration, reaching 0.9579 with mean squared error of 0.0004. Based on the experimental strain and acceleration results, the predicted fatigue lives of multiple linear regression models were correlated well with the experimental results with coefficient of determination value of 0.9275. Meanwhile, the maximum difference of vibration prediction to experimental value using multiple linear regression models was only 18%. For artificial neural network predictions, the fatigue lives were mostly distributed within 1:2 or 2:1 life correlation and vibration prediction results were within 12%. For a good prediction, the target correlation value was above 0.80 to demonstrate a good fitted curve and the difference below 20%. The trained artificial neural network has shown outstanding capability in fatigue life or ride-related vertical vibration predictions. In this research, the main novelty was the trained artificial neural network for spring fatigue life or ride-related vertical vibration predictions which serve to reduce some procedures of automotive suspension design. The outcome of this study can be used to provide a new knowledge towards the field of fatigue research as well as vehicle ride dynamics. This research contributes to automotive industries especially in suspension spring design where the analysis of fatigue and ride-related vibration are provided.

Machine learning based intelligent diagnosis methods can adaptively generate the fault diagnosis model by historical data, which have attracted much attention. Artificial neural network (ANN) is one of the most important tools for gearbox intelligent diagnosis. However, the training of ANN has the problem of local optima, and it is hard to determine the ANN structure. These problems have great influence on the diagnosis performance of ANN. In this paper, a variable neural network (RegPSOVNN) is proposed for gearbox fault diagnosis based on regrouping particle swarm optimization. Ten time-domain features are selected to form the input of the ANN. Regrouping particle swarm optimization (RegPSO) is utilized for the optimization of ANN structure and network training. It can simultaneously optimize the structure and parameters of ANN and effectively avoid the problem of local optima. To evaluate the diagnosis performance of the proposed method, gearbox failure experiments were conducted, and backpropagation neural network (BPNN), firefly variable neural network (FAVNN) and particle swarm optimization based variable neural network (PSOVNN) were used for comparison. Experimental results indicated that the proposed method can effectively optimize the network structure and diagnosis the gearbox faults.