Optimizing Learning Weights of Back Propagation Using Flower Pollination Algorithm for Diabetes and Thyroid Data Classification

The back propagation (BP) algorithm is the most popular supervised learning method for multi-layered feed forward Neural Network. It has been successfully deployed in numerous practical problems and disciplines. Regardless of its popularity, BP is still known for some major drawbacks such as easily getting stuck in local minima and slow convergence; since, it uses Gradient Descent (GD) method to learn the network. Over the years, many improved modifications of the BP learning algorithm have been made by researchers but the local minima problem remains unresolved. Therefore, to resolve the inherent problems of BP algorithm, this paper proposed BPGD-A3T algorithm where the approach introduces three adaptive parameters which are gain, momentum and learning rate in BP. The performance of the proposed BPGD-A3T algorithm is then compared with BPGD two term parameters (BPGD-2T), BP with adaptive gain (BPGD-AG) and conventional BP algorithm (BPGD) by means of simulations on classification datasets. The simulation results show that the proposed BPGD-A3T shows better performance and performed highest accuracy for all dataset as compared to other.

Mechanical sifting and manual grading have conventionally been utilised to assess the grade of aggregates. Nonetheless, such evaluations require a range of mechanical, chemical, and physical examinations, typically conducted manually, resulting in a process that is tedious, subjective, and labour-intensive. This research aims to provide an image-based classification system for the categorisation of aggregates. An artifi cial neural network (ANN) has been used to analyse the acquired images and categorise their shapes. The composite images are obtained and utilised as the input parameter for prediction prior to the thresholding step. The Log-sigmoid (Logsig) activation function, utilised in a Multilayer Perceptron (MLP) network, exhibits a lower mean square error (MSE) and superior regression performance relative to the Pureline activation functions. The Logsig-based network has a MSE of 1.7473 and a regression capability of 0.9521.

Solving Large Scale Classification Problems with Stochastic Based Optimization

Efficient Artificial neural networks based on a hybrid metaheuristic optimization algorithm for damage detection in laminated composite structures

Back Propagation (BP) algorithm is one of the oldest learning techniques used by artificial neural networks (ANN). It has successfully been implemented in various practical problems. However, the algorithm still faces some drawbacks such as getting easily stuck at local minima and needs longer time to converge on an acceptable solution. Recently, the introduction of Second Order Methods has shown a significant improvement on the learning in BP but it still has some drawbacks such as slow convergence and complexity. To overcome these limitations, this research proposed a modified approach for BP by introducing the Conjugate Gradient and Quasi-Newton which were Second Order methods together with ‘gain’ parameter. The performances of the proposed approach is evaluated in terms of lowest number of epochs, lowest CPU time and highest accuracy on five benchmark classification datasets such as Glass, Horse, 7Bit Parity, Indian Liver Patient and Lung Cancer. The results show that the proposed Second Order methods with ‘gain’ performed better than the BP algorithm.

Back Propagation (BP) is commonly used algorithm that optimize the performance of network for training multilayer feed-forward artificial neural networks. However, BP is inherently slow in learning and it sometimes gets trapped at local minima. These problems occur mailnly due to a constant and non-optimum learning rate (a fixed step size) in which the fixed value of learning rate is set to an initial starting value before training patterns for an input layer and an output layer. This fixed learning rate often leads the BP network towrds failure during steepest descent. Therefore to overcome the limitations of BP, this paper introduces an improvement to back propagation gradient descent with adapative learning rate (BPGD-AL) by changing the values of learning rate locally during the learning process. The simulation results on selected benchmark datasets show that the adaptive learning rate significantly improves the learning efficiency of the Back Propagation Algorithm

A Neural Network is one of the techniques by which we classify data. In this paper, we have proposed an effectively stacked autoencoder with the help of a modified sigmoid activation function. We have made a two-layer stacked autoencoder with a modified sigmoid activation function. We have compared our autoencoder to the existing autoencoder technique. In the existing autoencoder technique, we generally use the logsigmoid activation function. But in multiple cases using this technique, we cannot achieve better results. In that case, we may use our technique for achieving better results. Our proposed autoencoder may achieve better results compared to this existing autoencoder technique. The reason behind this is that our modified sigmoid activation function gives more variations for different input values. We have tested our proposed autoencoder on the iris, glass, wine, ovarian, and digit image datasets for comparison propose. The existing autoencoder technique has achieved 96% accuracy on the iris, 91% accuracy on wine, 95.4% accuracy on ovarian, 96.3% accuracy on glass, and 98.7% accuracy on digit (image) dataset. Our proposed autoencoder has achieved 100% accuracy on the iris, wine, ovarian, and glass, and 99.4% accuracy on digit (image) datasets. For more verification of the effeteness of our proposed autoencoder, we have taken three more datasets. They are abalone, thyroid, and chemical datasets. Our proposed autoencoder has achieved 100% accuracy on the abalone and chemical, and 96% accuracy on thyroid datasets.

This study discusses the negative impact of the derivative of the activation functions in the output layer of artificial neural networks, in particular in continual learning. We propose Hebbian descent as a theoretical framework to overcome this limitation, which is implemented through an alternative loss function for gradient descent we refer to as Hebbian descent loss. This loss is effectively the generalized log-likelihood loss and corresponds to an alternative weight update rule for the output layer wherein the derivative of the activation function is disregarded. We show how this update avoids vanishing error signals during backpropagation in saturated regions of the activation functions, which is particularly helpful in training shallow neural networks and deep neural networks where saturating activation functions are only used in the output layer. In combination with centering, Hebbian descent leads to better continual learning capabilities. It provides a unifying perspective on Hebbian learning, gradient descent, and generalized linear models, for all of which we discuss the advantages and disadvantages. Given activation functions with strictly positive derivative (as often the case in practice), Hebbian descent inherits the convergence properties of regular gradient descent. While established pairings of loss and output layer activation function (e.g., mean squared error with linear or cross-entropy with sigmoid/softmax) are subsumed by Hebbian descent, we provide general insights for designing arbitrary loss activation function combinations that benefit from Hebbian descent. For shallow networks, we show that Hebbian descent outperforms Hebbian learning, has a performance similar to regular gradient descent, and has a much better performance than all other tested update rules in continual learning. In combination with centering, Hebbian descent implements a forgetting mechanism that prevents catastrophic interference notably better than the other tested update rules. When training deep neural networks, our experimental results suggest that Hebbian descent has better or similar performance as gradient descent.

Gear and its transmission are widely used in different transmission systems, and its complicated and changeable condition brings a series of problems to the fault feature extraction and diagnosis. In recent years, deep learning techniques have been gradually applied to feature extraction and pattern recognition, and the features of feature extraction and fault diagnosis in complex working environments have shown certain advantages. This study is based on stacked autoencoder under deep learning model, and improve training network performance by modified activation function. Through the network training before and after the experiment done, and to extract the fault feature data comparison in testing, improving network after activation function to extract fault features showed a greater advantage, can be a very good application in practical fault feature extraction.

Artificial neural network (ANN) has been commonly used to deal with many problems. However, since this algorithm applies backpropagation algorithms based on gradient descent (GD) technique to look for the best solution, the network may face major risks of being entrapped in local minima. To overcome those drawbacks of ANN, in this work, we propose a novel ANN working parallel with metaheuristic algorithms (MAs) to train the network. The core idea is that first, (1) GD is applied to increase the convergence speed. (2) If the network is stuck in local minima, the capacity of the global search technique of MAs is employed. (3) After escaping from local minima, the GD technique is applied again. This process is applied until the target is achieved. Additionally, to increase the efficiency of the global search capacity, a hybrid of particle swarm optimization and genetic algorithm (PSOGA) is employed. The effectiveness of ANNPSOGA is assessed using both numerical models and measurement. The results demonstrate that ANNPSOGA provides higher accuracy than traditional ANN, PSO, and other hybrid ANNs (even a higher level of noise is employed) and also considerably decreases calculational cost compared with PSO.

The control of matte grade determines the production cost of the copper smelting process. In this paper, an optimal matte-grade control model is established to derive the optimal matte grade with the objective of minimizing the cost in the whole process of copper smelting. This paper also uses the prediction capability of the BP (Backpropagation) neural network to establish a BP neural network prediction model for the matte grade, considering various factors affecting matte grade (including the input copper concentrate amount and its composition content, air drumming amount, oxygen drumming amount, melting agent amount, and other process parameters). In addition, the paper also uses the optimal matte grade to optimize the dosing, air supply/oxygen supply, and oxygen supply for the ISA and other furnaces. When using BP networks only, it is a nonconvex problem with gradient descent, which tends to fall into local minima and has some bias in the prediction results. This problem can be solved by optimizing its weights and thresholds through GA (Genetic Algorithm) to find the optimal solution. The analysis results show that the average absolute error of the simulation of the BP neural network prediction model for ice copper grade after GA optimization is 0.51%, which is better than the average absolute error of 1.17% of the simulation of the single BP neural network model.

Multilayer perceptron (MLP) has been found to be most frequently used model for stock market forecasting. MLP is characterized with black-box in nature and lack of providing a formal method of deriving ultimate structure of the model. Higher order neural network (HONN) has the ability to expand the input representation space, perform high learning capabilities that require less memory in terms of weights and nodes and have been utilized in many complex data mining problems. To capture the extreme volatility, nonlinearity and uncertainty associated with stock data, this paper considered a HONN, called Pi-Sigma Neural Network (PSNN), for prediction of closing prices of five real stock markets. The tunable weights are optimized by Gradient Descent (GD) and a global search technique, Genetic Algorithm (GA). The model proves its superiority when trained with GA in terms of Average Percentage of Errors (APE).

Nowadays, the influence of machine learning algorithms on our lives is booming gradually, and gradient descent (GD) method is a typical big data analysis method that can be applied to various machine learning algorithms. Traditional GD method has its advantages in dealing with the optimal parameter finding issue for partial loss functions, but the optimal parameter finding process may be trapped into local minimum sometimes if the loss function with certain convexity; moreover, traditional GD cannot meet the requirement of high accuracy. Fractional order GD, which is the core of fixed memory step GD, has better performance by comparing with traditional GD, but still have some problems in the aspect of convergence. On the other hand, t-distributed machine learning algorithm for dimensionality reduction, can reduce the dimensionality of data in high-dimensional space to two-dimensional or three-dimensional space, and can convert the original data into more comprehensible images. In order to solve the problem mentioned above reasonably, in this paper, a fixed-memory step GD method combined with t-distributed stochastic neighbor embedding (t-SNE) is employed to replace the traditional GD method, so as to keep the consistency of low-dimensional data being mapped from high-dimensional data, and to achieve efficient data visualization. The effectiveness of proposed t-SNE algorithm based on fixed memory step gradient descent method has been verified finally via numerical experiment.

Recurrent neural network (RNN) has been widely used as a tool in the data classification. This network can be educated with gradient descent back propagation. However, traditional training algorithms have some drawbacks such as slow speed of convergence being not definite to find the global minimum of the error function since gradient descent may get stuck in local minima. As a solution, nature inspired metaheuristic algorithms provide derivative-free solution to optimize complex problems. This paper proposes a new metaheuristic search algorithm called Cuckoo Search (CS) based on Cuckoo bird’s behavior to train Elman recurrent network (ERN) and back propagation Elman recurrent network (BPERN) in achieving fast convergence rate and to avoid local minima problem. The proposed CSERN and CSBPERN algorithms are compared with artificial bee colony using BP algorithm and other hybrid variants algorithms. Specifically, some selected benchmark classification problems are used. The simulation results show that the computational efficiency of ERN and BPERN training process is highly enhanced when coupled with the proposed hybrid method.

BP network is one of the most popular artificial neural networks because of its special advantage such as simple structure, distributed storage, parallel processing, high fault-tolerance performance, etc. However, with its extensive use in recent years, it is discovered that BP algorithm has the defects on slow convergent speed and easy convergence to a local minimum point. The paper proposes a method of BP Neural Network improved by Particle Swarm Optimization (PSO). The hybrid algorithm can not only avoid local minimum, but also raise the speed of network training and reduce the convergence time.