Deterministic and Stochastic Logarithmic Barrier Function Methods for Neural Network Training

Error Correcting Output Codes, ECOC, is an output representation method capable of discovering some of the errors produced in classification tasks. This paper describes the application of ECOC to the training of feed forward neural networks, FFNN, for improving the overall accuracy of classification systems. Indeed, to improve the generalization of FFNN classifiers, this paper proposes an ECOC-Based training method for Neural Networks that use ECOC as the output representation, and adopts the traditional Back-Propagation algorithm, BP, to adjust weights of the network. Experimental results for face recognition problem on Yale database demonstrate the effectiveness of our method. With a rejection scheme defined by a simple robustness rate, high reliability is achieved in this application.

It is known that predictor-corrector methods in a large neighborhood of the central path are among the most efficient interior-point methods (IPMs) for linear optimization (LO) problems. The best iteration bound based on the classical logarithmic barrier function is O(nlog (n/∊)). In this paper, we propose a family of self-regular proximity-based predictor-corrector (SRPC) IPMs for LO in a large neighborhood of the central path. In the predictor step, we use either an affine scaling or a self-regular direction; in the corrector step, we use always a self-regular direction. Our new algorithms use a special proximity function with different search directions and thus allows us to improve the so far best theoretical iteration complexity for a family of SRPC IPMs. An O\((\sqrt{n}{\exp} ((1 - q + {\log} n)/2) {\log} n {\log} (n/\epsilon))\) worst-case iteration bound with quadratic convergence is established, where q is the barrier degree of the SR proximity function.

ADANOISE: Training neural networks with adaptive noise for imbalanced data classification

Automated retrosynthetic planning algorithms are a research area of increasing importance. Automated reaction-template extraction from large datasets, in conjunction with neural-network-enhanced tree-search algorithms, can find plausible routes to target compounds in seconds. However, the current method for training neural networks to predict suitable templates for a given target product leads to many predictions that are not applicable in silico. Most templates in the top 50 suggested templates cannot be applied to the target molecule to perform the virtual reaction. Here, we describe how to generate data and train a neural network policy that predicts whether templates are applicable or not. First, we generate a massive training dataset by applying each retrosynthetic template to each product from our reaction database. Second, we train a neural network to perform near-perfect prediction of the applicability labels on a held-out test set. The trained network is then joined with a policy model trained to predict and prioritize templates using the labels from the original dataset. The combined model was found to outperform the policy model in a route-finding task using 1700 compounds from our internal drug-discovery projects.

The paper presents a logarithmic barrier cutting plane algorithm for convex (possibly non-smooth, semi-infinite) programming. Most cutting plane methods, like that of Kelley, and Cheney and Goldstein, solve a linear approximation (localization) of the problem and then generate an additional cut to remove the linear program's optimal point. Other methods, like the “central cutting” plane methods of Elzinga-Moore and Goffin-Vial, calculate a center of the linear approximation and then adjust the level of the objective, or separate the current center from the feasible set. In contrast to these existing techniques, we develop a method which does not solve the linear relaxations to optimality, but rather stays in the interior of the feasible set. The iterates follow the central path of a linear relaxation, until the current iterate either leaves the feasible set or is too close to the boundary. When this occurs, a new cut is generated and the algorithm iterates. We use the tools developed by den Hertog, Roos and Terlaky to analyze the effect of adding and deleting constraints in long-step logarithmic barrier methods for linear programming. Finally, implementation issues and computational results are presented. The test problems come from the class of numerically difficult convex geometric and semi-infinite programming problems.

In this paper we introduce an advanced supervised training method for neural networks. It is based on Jacobian rank deficiency and it is formulated, in some sense, in the spirit of the Gauss-Newton algorithm. The Levenberg-Marquardt algorithm, as a modified Gauss-Newton, has been used successfully in solving nonlinear least squares problems including neural-network training. It outperforms (in terms of training accuracy, convergence properties, overall training time, etc.) the basic backpropagation and its variations with variable learning rate significantly, however, with higher computation and memory complexities within each iteration. The new method developed in this paper is aiming at improving convergence properties, while reducing the memory and computation complexities in supervised training of neural networks. Extensive simulation results are provided to demonstrate the superior performance of the new algorithm over the Levenberg-Marquardt algorithm.

In this study, performances comparison to discriminate five mental states of five artificial neural network (ANN) training methods were investigated. Wavelet Packet Transform (WPT) was used for feature extraction of the relevant frequency bands from raw electroencephalogram (EEG) signals. The five ANN training methods used were (a) Gradient Descent Back Propagation (b) Levenberg-Marquardt (c) Resilient Back Propagation (d) Conjugate Learning Gradient Back Propagation and (e) Gradient Descent Back Propagation with movementum.

Generation of the human anti-MUC1 peptide through neural network training and monomeric design method. Analyzing 9-mer peptide potential computationally for treatment of HER2-positive breast cancer. With the advancements of cancer genome atlas project (TCGA), cancer dependancy project (DepMap) and human protein atlas (HPA), large-scale datasets are generated for oncology studies. However, after development of redefined breast cancer drug targets, there are key issues in successful breast cancer treatments that needed to be pursued which paved the pathway for new approaches or strategies. In that respect, our research data aimed to represent a new aspect of breast cancer drug development studies. Extract human MUC1 sequences from various databases. Perform neural networking method for novel peptides sequences. Analyze the potentiality of generated heteroclitic peptide sequences for suitable vaccine candidate for breast cancer treatment. Input scaffolds of protein database (PDB) files for human MUC1 were retrieved and loaded into Evo design server with monomeric based design option. Further, neural network training approaches were followed and other computational tools were used for alignment-independent prediction of protective antigens and subunit vaccines potency of designed heteroclitic peptides. Study findings revealed two human anti-MUC1 heteroclitic peptides of 9mers (WAVWTYVSV, FMSFYIMNL), which showed the lowest energy cluster and sequence identity, normalized relative error rate of secondary structure, solvent accessibility, backbone torsion angles for neural networking and RMSD values in evolutionary profiling, and online MHCPred IC50 interaction values. VaxiGen v2.0 server revealed subunit vaccine potency values of in-silico designed two heteroclitic peptides were 0.1551 (WAVWTYVSV) and 0.3508 (FMSFYIMNL) with a threshold value of 0.5 followed by AllerTOP v2.0 for their allergenicity nature in immunogenic reactions. Computationally designed heteroclitic peptide WAVWTYVSV indicated promising values which can be utilised as drug delivery or tumour marker candidate in the treatment of human breast cancer by eliciting lyse of tumor cells.

In many real-world scenarios, labeled data for a specific machine learning task is costly to obtain. Semi-supervised training methods make use of abundantly available unlabeled data and a smaller number of labeled examples. We propose a new framework for semi-supervised training of deep neural networks inspired by learning in humans. Associations are made from embeddings of labeled samples to those of unlabeled ones and back. The optimization schedule encourages correct association cycles that end up at the same class from which the association was started and penalizes wrong associations ending at a different class. The implementation is easy to use and can be added to any existing end-to-end training setup. We demonstrate the capabilities of learning by association on several data sets and show that it can improve performance on classification tasks tremendously by making use of additionally available unlabeled data. In particular, for cases with few labeled data, our training scheme outperforms the current state of the art on SVHN.

Although the Levenberg-Marquardt (LM) algorithm has been extensively applied as a neural-network training method, it suffers from being very expensive, both in memory and number of operations required, when the network to be trained has a significant number of adaptive weights. In this paper, the behavior of a recently proposed variation of this algorithm is studied. This new method is based on the application of the concept of neural neighborhoods to the LM algorithm. It is shown that, by performing an LM step on a single neighborhood at each training iteration, not only significant savings in memory occupation and computing effort are obtained, but also, the overall performance of the LM method can be increased.

We study the properties of the weighted central paths in linear programming. We consider each path as the function of the parameter 0 where the value at D 0 corresponds to the limit point at the boundary of the feasible set. We calculate the recursive formulas for the central path derivatives of all orders valid at each 0. We establish the geometric growth of the derivatives and, consequently, the analyticity of the weighted central path at 0 . This paper also provides the analysis of limiting behavior of those projection operators which often appear in the interior point methods. The central trajectory is one of the most important concepts in the interior point methods for linear programming. The central trajectory (or the central path) is a distinguished curve that is interior to the feasible set and tends to an optimal point. Most interior points algorithms follow, or are related to, the central trajectory. Although the central path concept is essential for the design and analysis of algorithms relatively few papers study properties of this curve itself. For a review on this topic we refer to (3). Recently Zhao and Zhu (16) have studied some analytical properties of the central trajectory in the interior of the feasible set. They have established an upper bound for the high order derivatives of the weighted central path and estimated the convergence radius of the corresponding Taylor series. They have also pointed out the relationship of these properties with the complexity of a path following algorithm where the number of iterations is estimated by a "curvature integral". (See also (15).) Further, Guler (3) has analysed some limiting properties of the central trajectory derivatives with respect to the parameter >0. He has established the existence of finite limits for derivatives of all orders as tends to infinity (that is, the central path stays in the interior) and to zero (the central path tends to the boundary). These results have applications to the construction of polynomial- time algorithms which are also locally fast. A surprising result of this study is that all derivatives of order > 1t end to 0 as!1, which helps to explain why interior point methods for linear programming require so few iterations (see (3)).

The current state of scientific and technological development of the world economy is quite specific, because advanced technologies already known are too complicated for simple mechanical copying and borrowing, and most of the technologies of Industry 4.0 are in the making. Thus, the development and further exploitation of all kinds of innovations today, more than ever, require an appropriate environment - an effective national innovation system (NIS), which determines the country's ability to generate innovation, which is the key to high competitiveness and world leadership. However, the formation of a full-fledged innovation system of the country is quite complicated, for at least two reasons: first, there exist purely national features of functioning and cooperation of the main agents of change, and secondly, in the modern globalized world many of the most important for innovation processes go beyond the borders of individual countries, creating a unique transnational "ecosystem" with its distinctive features, which, undoubtedly, must be considered. The article proposes the scientific approach of reliable identification of national and transnational (supranational, global) innovation systems (TNIS) and the corresponding toolkit for simulating their development in the context of the quadruple helix concept. Identification of innovative systems is based on the methods of cluster analysis, genetic algorithms and neural network training. As a result, there have been identified and qualitatively interpreted four basic types of TNIS, which have stable characteristics determining the behavioural parameters and capabilities of the NIS included. A neural network has been built to identify NIS, which simplifies the process of simulating their development within the characteristic features of basic TNIS. It is established that the NIS of Ukraine belongs to the basic type of TNIS – “developed and developing countries with mixed extractive-inclusive institutions with a strong informal component (including the post-Soviet type)”. The results of its functioning against the background of global and relevant cluster leaders are not satisfactory and necessitate the adjustment of the further development vector. In order to demonstrate the capabilities of the neural network built, four supranational associations have been identified and analysed. The proposed approaches and tools will facilitate variant analytics and forecasting studies in substantiating the optimal directions for the individual NIS further development in the context of global and cluster trends.

This paper addresses the correct choice and combination of confidence measures in large vocabulary speech recognition tasks. We classify single words within continuous as well as large vocabulary utterances into two categories: utterances within the vocabulary which are recognized correctly, and other utterances, namely misrecognized utterances or (less frequent) out-of-vocabulary (OOV). To this end, we investigate the classification error rate (CER) of several classes of confidence measures and transformations. In particular, we employed data-independent and data-dependent measures. The transformations we investigated include mapping to single confidence measures and linear combinations of these measures. These combinations are computed by means of neural networks trained with Bayes-optimal, and with Gardner-Derrida-optimal criteria. Compared to a recognition system without confidence measures, the selection of (various combinations of) confidence measures, the selection of suitable neural network architectures and training methods, continuously improves the CER.

In the context of command-and-control applications, we exploit confidence measures in order to classify single-word utterances into two categories: utterances within the vocabulary which are recognized correctly, and other utterances, namely out-ofvocabulary (OOV) or misrecognized utterances. To this end, we investigate the classification error rate (CER) of several classes of confidence measures and transformations. In particular, we employed data-independent and data-dependent measures. The transformations we investigated include mapping to single confidence measures, LDA-transformed measures, and other linear combinations of these measures. These combinations are computed by means of neural networks trained with Bayesoptimal, and with Gardner-Derrida-optimal criteria. Compared to a recognition system without confidence measures, the selection of (various combinations of) confidence measures, the selection of suitable neural network architectures and training methods, continuously improves the CER. Additionally, we found that a linear perceptron generalizes better than a non-linear backpropagation network.

This study proposes a model that can be used for monitoring and identifying consumer electrical loads using Neural Network and Feed-Forward Backpropagation training methods. The modeling design is made in Simulink-Matlab software. The neural network training error obtained is 2.729%, then this is used for the identification process of electrical loads, which in this study used 21 load combinations. While the error test results obtained are 1.325476%. In this study, a new model is proposed for monitoring and identifying consumer electricity consumption by applying the concept of Non-Intrusive Load Monitoring (NILM) combined with the Artificial Neural Network (ANN) method.