Training Artificial Neural Networks by Generalized Likelihood Ratio Method: An Effective Way to Improve Robustness

<abstract><p>The training of artificial neural networks (ANNs) with rectified linear unit (ReLU) activation via gradient descent (GD) type optimization schemes is nowadays a common industrially relevant procedure. GD type optimization schemes can be regarded as temporal discretization methods for the gradient flow (GF) differential equations associated to the considered optimization problem and, in view of this, it seems to be a natural direction of research to <italic>first aim to develop a mathematical convergence theory for time-continuous GF differential equations</italic> and, thereafter, to aim to extend such a time-continuous convergence theory to implementable time-discrete GD type optimization methods. In this article we establish two basic results for GF differential equations in the training of fully-connected feedforward ANNs with one hidden layer and ReLU activation. In the first main result of this article we establish in the training of such ANNs under the assumption that the probability distribution of the input data of the considered supervised learning problem is absolutely continuous with a bounded density function that every GF differential equation admits for every initial value a solution which is also unique among a suitable class of solutions. In the second main result of this article we prove in the training of such ANNs under the assumption that the target function and the density function of the probability distribution of the input data are piecewise polynomial that every non-divergent GF trajectory converges with an appropriate rate of convergence to a critical point and that the risk of the non-divergent GF trajectory converges with rate 1 to the risk of the critical point. We establish this result by proving that the considered risk function is <italic>semialgebraic</italic> and, consequently, satisfies the <italic>Kurdyka-Łojasiewicz inequality</italic>, which allows us to show convergence of every non-divergent GF trajectory.</p></abstract>

Stochastic Gradient Estimation for Artificial Neural Networks

This paper presents a novel technique for the supervised training of feed-forward artificial neural networks (ANN) using the Harmony Search (HS) algorithm. HS is a stochastic meta-heuristic that is inspired from the improvisation process of musicians. Unlike Backpropagation, HS is non-trajectory driven. By modifying an existing improved version of HS & adopting a suitable ANN data representation, we propose a training technique where two of HS probabilistic parameters are determined dynamically based on the best-to-worst (BtW) harmony ratio in the current harmony memory instead of the improvisation count. This would be more suitable for ANN training since parameters and termination would depend on the quality of the attained solution. We have empirically tested and verified our technique by training an ANN with a benchmarking problem. In terms of overall training time and recognition, our results have revealed that our method is superior to both the original improved HS and standard Backpropagation.

The effects of data preprocessing in the determination of coagulant dosing rate

Selection of the most relevant features for training of an artificial neural network (ANN) is of significant importance affecting largely the complexity and accuracy of the trained network. Moreover, deciding the topology and architecture of an ANN including the training algorithm, number of hidden layers, activation functions and neurons in each layer is challenging for the effectiveness of an ANN to solve a particular problem. In this paper, we present a novel strategy to result into an optimized topological architecture for an ANN and selection method for the most important experimental features available for training. The proposed method is found to efficiently design and model flows in solar collectors and subsequently estimating pressure loss coefficients for flows in the tee junctions vastly used in solar collectors with high accuracy. Experimental results demonstrate that the number of features used in training of ANN can be reduced up to 85% by identifying the features upon which the pressure loss coefficients’ estimation shows strong dependency. Performance analysis of various training algorithms has been carried out with variation in hidden layers, activation functions and neurons that results into minimum mean square error leading to minimal percent relative error.

BACKGROUND: Currently, artificial intelligence in the form of artificial neural networks is being actively implemented in a number of areas of our lives, including medicine. In particular, in otorhinolaryngology, artificial neural networks are used to analyze images obtained during endoscopic examinations of patients (e.g., videolaryngoscopy) [1–3]. The interpretation of laryngoscopic images often presents significant difficulties for practicing physicians, which reduces the frequency of detection of precancerous laryngeal diseases and contributes to the increase in the number of patients with stage III–IV laryngeal cancer [4, 5]. This underscores the significance of prompt performance and accurate interpretation of the findings of endoscopic examinations of patients with laryngeal disorders. Artificial neural networks can be employed to analyze the results of videolaryngoscopy, furnishing the physician with supplementary information that can enhance diagnostic accuracy and diminish the probability of error [6, 7]. AIM: The study aims to develop and train an artificial neural network for recognizing characteristic features of laryngeal neoplasms and variants of laryngeal normality. MATERIALS AND METHODS: The study was conducted under the grant of the Moscow Center for Innovative Technologies in Healthcare (grant No. 2112-1/22) entitled “Using Neural Networks (Artificial Intelligence Algorithms) for Control and Improving the Quality of Diagnosis and Treatment of Diseases of Laryngeal and Ear Structures through Digital Technologies”.The following methods were used during the course of the study: data collection for the creation of a photobank (dataset) of medical images obtained during videolaryngoscopy; data partitioning for the formation of datasets for individual nosologies and groups of diseases; the method of consilium; analysis of the accuracy of recognition and classification of digital endoscopic images; and training of classification neural networks. Consequently, a dataset comprising 1,471 laryngeal images in digital formats (JPEG, BMP) was assembled, labelled, and uploaded for the purpose of training the artificial neural network. Of the total number of images, 410 were classified as pertaining to laryngeal formation, while 1061 were classified as variants of normality. Subsequently, the neural network was trained and tested to identify the signs of normal and laryngeal masses. RESULTS: The results of the testing of the artificial neural network indicated the formation of an inaccuracy matrix, the calculation of the value of recognition accuracy, the calculation of the quality indicators of the model performance, and the construction of the ROC curve. The developed and trained artificial neural network demonstrated an accuracy of 86% in recognizing the signs of laryngeal masses and norms. CONCLUSIONS: This study demonstrates that a trained artificial neural network can successfully distinguish between signs of normal and laryngeal masses in endoscopic photographs. With further training of the neural network and achievement of high accuracy, this technology can be used in clinical practice as an assistant in the interpretation of laryngoscopic images and early diagnosis of laryngeal masses. It can also be employed to control and improve the quality of diagnosis and treatment of diseases of the throat, nose, and ears by primary care physicians.

In this work a modification was made on the algorithm of Artificial Neural Networks (NN) Training of the Multilayer Perceptron type (MLP) based on multi-objective optimization (MOBJ), to increase its computational efficiency. Usually, the number of efficient solutions to be generated is a parameter that must be provided by the user. In this work, this number is automatically determined by an algorithm, through the usage of golden section, being generally less when specified, showing a sensible reduction in the processing time and keeping the high generalization capability of the obtained solution from the original method.

In this paper, we show the possibility of creating and identifying the features of an artificial neural network (ANN), which consists of mathematical models of biological neurons. The FitzHugh–Nagumo (FHN) system is used as a paradigmatic model demonstrating basic neuron activities. First, in order to reveal how biological neurons can be embedded within an ANN, we train the ANN with nonlinear neurons to solve a basic image recognition problem with an MNIST database; next, we describe how FHN systems can be introduced into this trained ANN. After all, we show that an ANN with FHN systems inside can be successfully trained with improved accuracy comparing with first trained ANN and then with inserted FHN systems. This approach opens up great opportunities in terms of the direction of analog neural networks, in which artificial neurons can be replaced by more appropriate biological ones.

In this work, a methodology was developed to estimate the mechanical properties (Brinell hardness, ultimate tensile strength and yield strength) using instrumented macroindentation tests. This methodology was based on the training of artificial neural networks (ANNs) from experimental curves of spherical indentations. Different ANNs architectures were implemented and trained with three algorithms, namely: a) genetic algorithm (GA); b) hybrid algorithm between GA and Levenberg- Marquardt (LM) method with Bayesian regularization (BR), named GA-LMBR; and c) the GA-LMBR method, with the addition of a similarity verification step among GA individuals. The input patterns of the ANNs were obtained from the materials tests performed by Nicolosi (2015) through a portable macroindentator called PropInSitu 2. Among the results obtained, it was verified that the hybrid algorithm GA-LMBR provided the best results, i.e., it resulted in the smallest errors in the estimation of the mechanical properties. These results were achieved when the GA-LMBR method was applied to an ANN composed of a nine-variable input layer, a hidden layer with two neurons and an output layer with one neuron. In addition, certain values of GA performance parameters, such as the number of individuals of the initial population, stopping criterion, crossover rate, mutation rate, among others; and certain neuron activation functions were used. The activation functions employed in the hidden neurons and the output neuron were, respectively, sigmoid and linear. The methodology developed was efficient in determining the mechanical properties, since the best results provided small errors compared to the traditional methods. Considering 95 % confidence level, the errors for the Brinell hardness estimation were in the range of ±3 %, for the ultimate tensile strength estimation, in the range of ±6 % and for the yield strength estimation, in the range of ±8 %.

The aim of this article is to use artificial neural networks (ANN) to perform the structural design of confined masonry buildings. ANN is easy to operate and allows to reduce the time and cost of seismic designs. To generate the artificial neural network, training models (traditional confined masonry designs) are used to identify the input and output parameters. From this, the final architecture and activation functions are defined for each layer of the ANN. Finally, ANN training is carried out using the "backpropagation" algorithm to obtain the matrix of weights and thresholds that allow the network to operate and provide preliminary structural designs with a 10% margin of error, with respect to the traditional design, in the dimensions and reinforcements of the structural elements.

Providing the real-time working system able to localize the dangerous contaminant source is one of the main challenges of the cities emergency response groups. Unfortunately, all proposed up to now frameworks capable of estimating the contamination source localization based on recorded by the sensors network the substance concentrations are not able to work in real-time. The reason is the significant computational time required by the applied dispersion models. In such reconstruction systems, the parameters of the given dispersion model are sampled to fit the model output to the registrations; thus, the dispersion model is run tens of thousands of times. In this paper, we test the possibility of training an artificial neural network (ANN) to effectively simulate the atmospheric toxin transport in the highly urbanized area. The use of a fast neural network in place of computationally costly dispersion models in systems localizing the source of contamination can enable its fast response time. As a training domain, we have chosen the center of London, as it was used in the DAPPLE field experiment. The training dataset is generated by the Quick Urban & Industrial Complex (QUIC) Dispersion Modeling System. To achieve the ANN capable of estimating the contaminant concentration, we tested various ANN structures, i.e., numbers of ANN layers, neurons, and activation functions. The performed tests confirm that trained ANN has the potential to replace the dispersion model in the contaminant source localization systems.

A novel approach to accelerate training in artificial neural network to detect three-phase flows using a gamma source and a detector

1176 OUTCOME PREDICTION IN PATIENTS WITH BLADDER CANCER AFTER RADICAL CYSTECTOMY USING ARTIFICIAL NEURAL NETWORKS

Spiking neural networks (SNNs) more closely mimic the human brain than artificial neural networks (ANNs). For SNNs, time-to-first-spike (TTFS) encoding, which represents the output values of neurons based on the timing of a single spike, has been proposed as a promising model to reduce power consumption. Adversarial attacks that can lead ANNs to misrecognize images have been reported in many studies. However, the characteristics of TTFS-based SNNs trained using a backpropagation algorithm against adversarial attacks have not yet been clarified. In particular, the dependence of the robustness against adversarial attacks on spike timings has not been investigated. In this brief, we investigated the robustness of SNNs against adversarial attacks and compared it with that of an ANN. We found that SNNs trained with the appropriate temporal penalty settings are more robust against adversarial images than ANNs.

Looking at the artificial neural networks’ literature, most of the studies started with feedforward artificial neural networks and the training of many feedforward artificial neural networks models are performed with derivative-based algorithms such as levenberg–marquardt and back-propagation learning algorithms in the first studies. In recent years, although many new heuristic algorithms have been proposed for different aims these heuristic algorithms are also frequently used in the training process of many different artificial neural network models. Pi-sigma artificial neural networks have different importance than many artificial neural network models with its higher-order network structure and superior forecasting performance. In this study, the training of Pi-Sigma artificial neural networks is performed by differential evolution algorithm uses DE/rand/1 mutation strategy. The performance of the proposed method is evaluated by two data sets and seen that the proposed method has a very effective performance compared with many artificial neural network models.