Advanced neural-network training algorithm with reduced complexity based on Jacobian deficiency

Error Correcting Output Codes, ECOC, is an output representation method capable of discovering some of the errors produced in classification tasks. This paper describes the application of ECOC to the training of feed forward neural networks, FFNN, for improving the overall accuracy of classification systems. Indeed, to improve the generalization of FFNN classifiers, this paper proposes an ECOC-Based training method for Neural Networks that use ECOC as the output representation, and adopts the traditional Back-Propagation algorithm, BP, to adjust weights of the network. Experimental results for face recognition problem on Yale database demonstrate the effectiveness of our method. With a rejection scheme defined by a simple robustness rate, high reliability is achieved in this application.

ADANOISE: Training neural networks with adaptive noise for imbalanced data classification

In this paper, we use block-diagonal matrix of approximate the Hessian matrix in the Levenberg Marquardt method during the training of recurrent neural networks. We analyze the weight updating strategies and the groupings of the weights associated with the approximation. Two weight updating strategies, namely asynchronous and synchronous updating methods are investigated. Asynchronous method updated weights of one block at a time while synchronous method updates all weights at the same time. Variations of these two methods, which involve the determination of two parameters mu and lambda, are examined. Four weight grouping methods, correlation blocks, k-unit blocks, layer blocks and arbitrary blocks are investigated and compared. Their computational complexity, approximation ability, and training time is analyzed. Comparing with the original Levenberg Marquardt method, the block-diagonal approximation methods give substantial improvement in training time without degrading the generalization ability.

Although the Levenberg-Marquardt (LM) algorithm has been extensively applied as a neural-network training method, it suffers from being very expensive, both in memory and number of operations required, when the network to be trained has a significant number of adaptive weights. In this paper, the behavior of a recently proposed variation of this algorithm is studied. This new method is based on the application of the concept of neural neighborhoods to the LM algorithm. It is shown that, by performing an LM step on a single neighborhood at each training iteration, not only significant savings in memory occupation and computing effort are obtained, but also, the overall performance of the LM method can be increased.

A homotopy method for training neural networks

In many real-world scenarios, labeled data for a specific machine learning task is costly to obtain. Semi-supervised training methods make use of abundantly available unlabeled data and a smaller number of labeled examples. We propose a new framework for semi-supervised training of deep neural networks inspired by learning in humans. Associations are made from embeddings of labeled samples to those of unlabeled ones and back. The optimization schedule encourages correct association cycles that end up at the same class from which the association was started and penalizes wrong associations ending at a different class. The implementation is easy to use and can be added to any existing end-to-end training setup. We demonstrate the capabilities of learning by association on several data sets and show that it can improve performance on classification tasks tremendously by making use of additionally available unlabeled data. In particular, for cases with few labeled data, our training scheme outperforms the current state of the art on SVHN.

Second-order optimization methods for training neural networks, such as KFAC, exhibit superior convergence by utilizing curvature information of loss landscape. However, it comes at the expense of high computational burden. In this work, we analyze the two components that constitute the layer-wise Fisher information matrix (FIM) used in KFAC: the Kronecker factors related to activations and pre-activation gradients. Based on empirical observations on their eigenspectra, we propose efficient approximations for them, resulting in a computationally efficient optimization method called MAC. To the best of our knowledge, MAC is the first algorithm to apply the Kronecker factorization to the FIM of attention layers used in transformers and explicitly integrate attention scores into the preconditioning. We also study the convergence property of MAC on nonlinear neural networks and provide two conditions under which it converges to global minima. Our extensive evaluations on various network architectures and datasets show that the proposed method outperforms KFAC and other state-of-the-art methods in terms of accuracy, end-to-end training time, and memory usage.

We describe an online method of training neural networks, which is based on solving the linearized least-squares problem using the pseudo-inverse for the underdetermined case. This underdetermined linearized least squares (ULLS) method requires significantly less computation and memory for implementation than standard higher-order methods such as the Gauss-Newton method or extended Kalman filter. This decrease is possible because the method allows training to proceed with a smaller number of samples than parameters. Simulation results which compare the performance of the ULLS algorithm to the recursive linearized least squares algorithm (RLLS) and the gradient descent algorithm are presented. Results showing the impact on computational complexity and squared-error performance of the ULLS method, when the number of terms in the Jacobian matrix is varied, are also presented.

This study proposes a model that can be used for monitoring and identifying consumer electrical loads using Neural Network and Feed-Forward Backpropagation training methods. The modeling design is made in Simulink-Matlab software. The neural network training error obtained is 2.729%, then this is used for the identification process of electrical loads, which in this study used 21 load combinations. While the error test results obtained are 1.325476%. In this study, a new model is proposed for monitoring and identifying consumer electricity consumption by applying the concept of Non-Intrusive Load Monitoring (NILM) combined with the Artificial Neural Network (ANN) method.

Automated retrosynthetic planning algorithms are a research area of increasing importance. Automated reaction-template extraction from large datasets, in conjunction with neural-network-enhanced tree-search algorithms, can find plausible routes to target compounds in seconds. However, the current method for training neural networks to predict suitable templates for a given target product leads to many predictions that are not applicable in silico. Most templates in the top 50 suggested templates cannot be applied to the target molecule to perform the virtual reaction. Here, we describe how to generate data and train a neural network policy that predicts whether templates are applicable or not. First, we generate a massive training dataset by applying each retrosynthetic template to each product from our reaction database. Second, we train a neural network to perform near-perfect prediction of the applicability labels on a held-out test set. The trained network is then joined with a policy model trained to predict and prioritize templates using the labels from the original dataset. The combined model was found to outperform the policy model in a route-finding task using 1700 compounds from our internal drug-discovery projects.

In this study, performances comparison to discriminate five mental states of five artificial neural network (ANN) training methods were investigated. Wavelet Packet Transform (WPT) was used for feature extraction of the relevant frequency bands from raw electroencephalogram (EEG) signals. The five ANN training methods used were (a) Gradient Descent Back Propagation (b) Levenberg-Marquardt (c) Resilient Back Propagation (d) Conjugate Learning Gradient Back Propagation and (e) Gradient Descent Back Propagation with movementum.

Historical database training method for neural networks

Logarithmic barrier methods are developed for training supervised neural networks. Two algorithms are presented: a deterministic logarithmic barrier and a stochastic logarithmic barrier. Specifically, we consider neural network training as a nonlinear programming problem and use barrier methods to find the optimal weights. Furthermore, we put constraints on the weights to avoid network paralysis. The search direction is derived using a recursive prediction error method (RPEM) that approximates the inverse of the Hessian of a logarithmic error function iteratively. The weights move on a center trajectory in the interior of the feasible weight space and have good convergence properties. In the case of the stochastic version, random fluctuations are added to the weights in order to escape local minima. Ill-conditioning problems are discussed and computational experiments are provided.Keywordsneural networksdeterministic and stochastic logarithmic barrier algorithmsinterior point methodsNewton type methodsneural network trainingANN training parallelization

This paper deals with a method for training neural networks by using cellular genetic algorithms (CGA). This method was implemented as software, CGANN-Trainer, which was used to generate binary classifiers for recognition of patterns associated with breast cancer images in a multi-objective optimization problem. The results reached by the CGA with the Wisconsin Breast Cancer Database, and the Wisconsin Diagnostic Breast Cancer Database, were compared with some other methods previously reported using the same databases, proving to be an interesting alternative.KeywordsNeural networksgenetic algorithmscellular automatamulti-objective classification

Generation of the human anti-MUC1 peptide through neural network training and monomeric design method. Analyzing 9-mer peptide potential computationally for treatment of HER2-positive breast cancer. With the advancements of cancer genome atlas project (TCGA), cancer dependancy project (DepMap) and human protein atlas (HPA), large-scale datasets are generated for oncology studies. However, after development of redefined breast cancer drug targets, there are key issues in successful breast cancer treatments that needed to be pursued which paved the pathway for new approaches or strategies. In that respect, our research data aimed to represent a new aspect of breast cancer drug development studies. Extract human MUC1 sequences from various databases. Perform neural networking method for novel peptides sequences. Analyze the potentiality of generated heteroclitic peptide sequences for suitable vaccine candidate for breast cancer treatment. Input scaffolds of protein database (PDB) files for human MUC1 were retrieved and loaded into Evo design server with monomeric based design option. Further, neural network training approaches were followed and other computational tools were used for alignment-independent prediction of protective antigens and subunit vaccines potency of designed heteroclitic peptides. Study findings revealed two human anti-MUC1 heteroclitic peptides of 9mers (WAVWTYVSV, FMSFYIMNL), which showed the lowest energy cluster and sequence identity, normalized relative error rate of secondary structure, solvent accessibility, backbone torsion angles for neural networking and RMSD values in evolutionary profiling, and online MHCPred IC50 interaction values. VaxiGen v2.0 server revealed subunit vaccine potency values of in-silico designed two heteroclitic peptides were 0.1551 (WAVWTYVSV) and 0.3508 (FMSFYIMNL) with a threshold value of 0.5 followed by AllerTOP v2.0 for their allergenicity nature in immunogenic reactions. Computationally designed heteroclitic peptide WAVWTYVSV indicated promising values which can be utilised as drug delivery or tumour marker candidate in the treatment of human breast cancer by eliciting lyse of tumor cells.