Simple, Efficient and Convenient Decentralized Multi-task Learning for Neural Networks

Various learning methods of neural networks including supervised and unsupervised methods are presented and illustrated with examples. General learning rule as a function of the incoming signals is discussed. Other learning rules such as Hebbian learning, perceptron learning, LMS (least mean square) learning, delta learning, WTA (winner take all) learning, and PCA (principal component analysis) are presented as a derivation of the general learning rule. Architecture specific learning algorithms for cascade correlation networks, Sarajedini and Hecht-Nielsen networks, functional link networks, polynomial networks, counterpropagation networks, RBF (radial basis function) networks are described. Dedicated learning algorithms for on chip neural network training are also evaluated. The tutorial focuses on various practical methods such as Quickprop, RPROP, Back Percolation, Delta-bar-Delta and others. Main reasons of convergence difficulties such as local minima or flat spot problems are analyzed. More advance gradient-based methods including pseudo inversion learning, conjugate gradient, Newton and LM (Levenberg-Marquardt) algorithm are illustrated with examples.

Currently, network security has become a major concern for all entities around the world. Attackers employ various methods to disrupt services, which requires new methods to stop them all in one way. Moreover, these intrusions can evolve and overcome security measures and devices, which pushes to use new evolving methods able to accompany the evolution of these threats, to block them. In our paper, we propose a new approach for intrusion detection, founded on neural network (NN) and deep learning (DL) methods. This approach is planned to not only identify threats, but also to develop a long-term memory of them, in order to detect new ones resembling these memorized attacks, and simultaneously, to provide a single way to stop all kinds of intrusions. To test our model, we have chosen the most recently employed methods in literature, NN and DL algorithms: feedforward neural network (FNN), convolutional neural network (CNN), and long short-term memory (LSTM), then we have applied them on network security layer-knowledge discovery in databases (NSL KDD) intrusions dataset. The results of experiments were impressive for all the algorithms, with maximum performances noted by LSTM, which affirms the efficacy of our proposed method for intrusion detection.

Highlights Convolutional neural network and MMD transfer learning methods are applied in inversion of optical properties. The classification accuracy of apples’ peel and pulp absorption coefficients are 84.61% and 92.47%, the accuracy of peel and pulp scattering coefficients are 83.56% and 86.53%, respectively. The depth optical characteristics can better reflect brix and moisture of apple then optical properties and hyperspectral data, the correlations are in the form of 0.98 and 0.98. Abstract. An inversion of optical properties is an important test for determining the quality of fruit. The conventional inversion model of the optical properties uses measured hyperspectral images as the training data. Studies show that the conventional machine learning method for inverting the optical properties results in low inversion accuracy, especially with curved models. Hence, the present study uses a convolutional neural network scheme to train the simulated hyperspectral images. Moreover, the maximum mean discrepancy (MMD) transfer method is used to transfer the simulated hyperspectral images to the measured hyperspectral images of apples. To evaluate the performance of the proposed method, the present study uses it to classify a variety of an apple’s optical properties, including the peel absorption, pulp absorption, peel scattering, and pulp scattering coefficients. The classification accuracies of the peel and pulp absorption coefficients are 84.61% and 92.47%, respectively. The classification accuracies of the peel and pulp scattering coefficients are 83.56% and 86.53%, respectively. These inversion results are compared with convolutional neural networks, neural networks, and support vector machines with measured hyperspectral images. It was found that the proposed inversion model is an effective scheme for optical property inversion. To prove the necessity of optical property inversion, the least squares, decision tree and random forest regression methods are performed to analyze the correlation between the depth of optical characteristics and the brix and moisture. The present study shows that these correlations are in the form of 0.98 and 0.98. The correlation coefficients increase by 0.36 and 0.25 compared to the measured hyperspectral images. The conclusions show that the proposed inversion model is an effective scheme for apple optical property inversion. Keywords: Apple tissue, Hyperspectral, Optical property inversion, Quality inspection.

Many real scenarios in machine learning are non-stationary. These challenges forces to develop new algorithms that are able to deal with changes in the underlying problem to be learnt. These changes can be gradual or abrupt. As the dynamics of the changes can be different, the existing machine learning algorithms exhibit difficulties to cope with them. In this work we propose a new method, that is based in the introduction of a forgetting function in an incremental online learning algorithm for two-layer feedforward neural networks. This forgetting function gives a monotonically crescent importance to new data. Due to this fact, the network forgets in presence of changes while maintaining a stable behavior when the context is stationary. The theoretical basis for the method is given and its performance is illustrated by evaluating its behavior. The results confirm that the proposed method is able to work in evolving environments.

The frequency of armed conflicts increased during the last 20 years. The problems of the emergence of military disputes, not only concern social parameters, but also economic and financial dimensions. This study examines the potential impact of global geopolitical events on the stock market prices of the Dow Jones U.S. Aerospace & Defense Index and Foreign Exchange (FOREX) markets movements. We analyse whether defence stocks and exchange rate perform similarly during military incidents or geopolitical crises. We built an Autoregressive Moving Average Model with a Generalized Autoregressive Conditional Heteroskedasticity process (ARMA-GARCH) with the machine learning methods of Neural Networks, Deep Recurrent Convolutional Neural Networks, Deep Neural Decision Trees, Quantum Neural Networks, and Quantum Recurrent Neural Networks, aimed at detecting intraday patterns for forecasting defence stock market and FOREX markets disturbances in a market microstructure framework. The empirical results provide preliminary findings on the foreseeability of market disturbances and small differences are observed before and during geopolitical events. Additionally, we confirm the effectiveness of the hybrid model ARMA-GARCH with the machine learning approaches, being ARMA-GARCH-Quantum Recurrent Neural Network the technique that achieves the best accuracy results. Our work has a large potential impact on investment market agents and portfolio managers, as shocks from geopolitical events could provide a new methodology to support the decision-making process for trading in High-Frequency Trading.

This paper presents the deduction of the quasi-Newton learning methods for training quaternion-valued feedforward neural networks, using the framework of the HR calculus. Since these algorithms yielded better training results than the gradient descent for the real- and complex-valued cases, an extension to the quaternion-valued case is a natural idea to enhance the performance of quaternion-valued neural networks. Experiments done on four time series prediction applications show a significant improvement over the quaternion gradient descent algorithm.

Neural networks provide a solution to overcome drawbacks of the quantitative fault diagnosis because first, they are capable to model off-line the behaviour of linear and non-linear systems. Secondly, they can also learn on-line the behaviour of a system requiring no priori knowledge about the system. The neural networks are particularly good for fault diagnosis of systems that have imperfect models and/or incomplete data. There are two basic learning methods of neural networks that are applied to fault diagnosis: supervised and unsupervised learning methods. To solve problem of priori unknown faults, unsupervised learning is used. The Counterpropagation network was selected to diagnose faults as result of analysis of supervised and unsupervised learning methods applied to neural networks.

This study uses a machine learning algorithm which is the extreme model one of the new learning method of neural networks. Determining the value of forecasting based on actual value. Output generated more quickly in this process, because the learning done in a fast speed and better accuracy rate than conventional forecasting methods. Epoch using the parameter changes and changes in the range of accuracy of test results demonstrate the value of error is quite good during the system testing.

Machine Learning in Engineering Analysis and Design: An Integrated Fuzzy Neural Network Learning Model

The Sixth International Symposium on Neural Networks (ISNN 2009)

This work mainly aimed to identify the most suitable calculation approaches and methods for the effect of inputs upon the outputs of a functionally-mathematical library with inaccessible source code, and further use these methods and approaches to develop an input ranking tool. These methods and approaches were studied using a generalized regressive model representing an arbitrary functionally-mathematical library (an executable module). The paper studied two approaches to the determination of input effects upon the outputs: 1) correlation analysis; 2) neural network method. The first method is analytical calculation (couple by couple) of Pierson correlation coefficients for all input and output parameters. As an alternative, the study also investigated rank-based Spierman and Kendall correlations. The second approach basically meant neural network learning with required accuracy using the variation set of executable module inputs and their respective outputs with subsequent calculation of contribution from each parameter to the overall result for the neural network after learning. These approaches have also been compared in terms of several major criteria, i.e. application peculiarities, speed and output quality. The analysis has shown that neural-network learning method, despite its certain drawbacks, is more suitable for the task of this study. The paper also outlined possible ways for further improvement of this method and complexity increase of the suggested functionally-mathematical library model.

With the popularity of smart wearable systems, sensor signal processing poses more challenges to machine learning in embedded scenarios. For example, traditional machine-learning methods for data classification, especially in real time, are computationally intensive. The deployment of Artificial Intelligence algorithms on embedded hardware for fast data classification and accurate fall detection poses a huge challenge in achieving power-efficient embedded systems. Therefore, by exploiting the associative memory feature of Hopfield Neural Network, a hardware module has been designed to simulate the Neural Network algorithm which uses sensor data integration and data classification for recognizing the fall. By adopting the Hebbian learning method for training neural networks, weights of human activity features are obtained and implemented/embedded into the hardware design. Here, the neural network weight of fall activity is achieved through data preprocessing, and then the weight is mapped to the amplification factor setting in the hardware. The designs are checked with validation scenarios, and the experiment is completed with a Hopfield neural network in the analog module. Through simulations, the classification accuracy of the fall data reached 88.9% which compares well with some other results achieved by the software-based machine-learning algorithms, which verify the feasibility of our hardware design. The designed system performs the complex signal calculations of the hardware’s feedback signal, replacing the software-based method. A straightforward circuit design is used to meet the weight setting from the Hopfield neural network, which is maximizing the reusability and flexibility of the circuit design.

In this chapter, first we introduce the basic concepts and models for neural networks and deep learning methods. Then we provide a survey on recent and emerging literature on deep learning methods for EHR data analysis, which include deep convolutional neural network (CNN), recurrent neural network (RNN), and generative adversarial network (GAN). We discuss the pros and cons of these methods with applications to EHR data analysis and prediction. We also share our experience in implementing these machine learning methods using different software tools such as TensorFlow, Torch/Pytorch, CNTK, Keras, Caffe, Theano, MXNet, and H2O. Two application examples with analysis scripts are presented to demonstrate the usefulness of neural networks and deep learning techniques for EHR data analysis and prediction.

Presents a new hybrid system which integrates a neural network and fuzzy rule-based system learning methods. The data used in this study were collected from 100 women who were clinically diagnosed with breast cancer in the form of carcinoma or benign conditions. The data set contains seven different histological and cytological factors, and two nodal outputs (positive and negative nodal status) to be predicted for nodal involvement assessment in breast cancer patients. The hybrid system yielded the highest predictive accuracy of 73%, compared with statistical, neural networks and fuzzy logic methods. The overall results are encouraging and reveal the efficiency of the hybrid system.

Accurately forecasting the daily production of coalbed methane (CBM) is important forformulating associated drainage parameters and evaluating the economic benefit of CBM mining. Daily production of CBM depends on many factors, making it difficult to predict using conventional mathematical models. Because traditional methods do not reflect the long-term time series characteristics of CBM production, this study first used a long short-term memory neural network (LSTM) and transfer learning (TL) method for time series forecasting of CBM daily production. Based on the LSTM model, we introduced the idea of transfer learning and proposed a Transfer-LSTM (T-LSTM) CBM production forecasting model. This approach first uses a large amount of data similar to the target to pretrain the weights of the LSTM network, then uses transfer learning to fine-tune LSTM network parameters a second time, so as to obtain the final T-LSTM model. Experiments were carried out using daily CBM production data for the Panhe Demonstration Zone at southern Qinshui basin in China. Based on the results, the idea of transfer learning can solve the problem of insufficient samples during LSTM training. Prediction results for wells that entered the stable period earlier were more accurate, whereas results for types with unstable production in the early stage require further exploration. Because CBM wells daily production data have symmetrical similarities, which can provide a reference for the prediction of other wells, so our proposed T-LSTM network can achieve good results for the production forecast and can provide guidance for forecasting production of CBM wells.