A review of automatic differentiation and its efficient implementation

Artificial neural networks (ANN) have been widely used in the field of data classification. Normally, training of neural network is applied with the traditional back propagation technique. As, this approach has various drawbacks, training of neural network is done with Particle Swarm Optimization (PSO). PSO has been widely used to solve the diverse kind of optimization problems. Population initialization performs a significant role in meta-heuristic algorithms. This paper describes a new initialization population approach Log Logistic termed as PSOLL-NN to create the initialization of the swarm. The proposed algorithm has been tested for weight optimization of feed forward neural network; and compared with back propagation Algorithm (BPA), standard PSO (PSO-NN), PSO initialized with Halton Sequence (PSOH-NN), Torus sequence (PSOT-NN) and Sobol sequence (PSOS-NN). The experimental results show that the proposed technique performed exceptionally better than the other traditional techniques. Moreover, the outcome of our work presents a foresight that how the proposed initialization technique can be used as an efficient alternative to standard training approaches for the data classification problems.

Traditional neural network training is usually based on the maximum likelihood to obtain the appropriate network parameters including weights and biases given the training data. However, if the available data are finite and noisy, the maximum likelihood-based network training can cause the neural network after being trained to overfit the noisy data. This problem has been overcome by using the Bayesian inference applied to the neural network training in various applications. The Bayesian inference can allow values of regularization parameters to be found using only the training data. In addition, the Bayesian approach also allows different models (e.g., neural networks with different numbers of hidden units to be compared using only the training data). Neural networks trained with Bayesian inference can be also known as Bayesian neural networks (BNNs). This chapter focuses on BNNs for classification problems with considerations of model complexity of BNNs conveniently handled by a method known as the evidence framework.

Expression templates are a well-known set of techniques for improving the efficiency of operator overloading-based forward mode automatic differentiation schemes in the C\(++\) programming language by translating the differentiation from individual operators to whole expressions. However standard expression template approaches result in a large amount of duplicate computation, particularly for large expression trees, degrading their performance. In this paper we describe several techniques for improving the efficiency of expression templates and their implementation in the automatic differentiation package Sacado (Phipps et al., Advances in automatic differentiation, Lecture notes in computational science and engineering, Springer, Berlin, 2008; Phipps and Gay, Sacado automatic differentiation package. http://trilinos.sandia.gov/packages/sacado/, 2011). We demonstrate their improved efficiency through test functions as well as their application to differentiation of a large-scale fluid dynamics simulation code.

Models of the diffusion-weighted signal are of strong interest for population studies of the brain microstructure. These studies are typically conducted by extracting a scalar property from the model and subjecting it to null hypothesis significance testing. This process has two major limitations: the reported p-value is a weak predictor of the reproducibility of findings and evidence for the absence of microstructural alterations cannot be gained. To overcome these limitations, this paper proposes a Bayesian framework for population studies of the brain microstructure represented by multi-fascicle models. A hierarchical model is built over the biophysical parameters of the microstructure. Bayesian inference is performed by Hamiltonian Monte Carlo sampling and results in a joint posterior distribution over the latent microstructure parameters for each group. Inference from this posterior enables richer analyses of the brain microstructure beyond the dichotomy of significance testing. Using synthetic and in-vivo data, we show that our Bayesian approach increases reproducibility of findings from population studies and opens new opportunities in the analysis of the brain microstructure.

This work deals with an implementation of automatic differentiation of C++ computer programs in forward mode using operator overloading and expression templates. We report on the efficiency of such implementation and its obvious advantage : the ability to perform sensitivity analysis without touching the source of the computer program by simply adding a library to it. We apply this tool to a flow control problem : minimize the drag of a cylinder, in subsonic unsteady turbulent flow, by controlling the boundary condition of the cylinder.

Proposes a fuzzy neural expert system (FNES) which has a feedforward fuzzy neural network whose input layer consists of fuzzy cell groups and crisp (non-fuzzy) cell groups. Here, the truthfulness of fuzzy information and crisp information of training data is represented by fuzzy cell groups and crisp cell groups, respectively. The expert system has the following two functions: generalization of the information derived from the training data and embodiment of knowledge in the form of the fuzzy neural network; and extraction of fuzzy If-Then rules with linguistic relative importance of each proposition in an antecedent (If-part) from a trained fuzzy neural network. The paper also gives a method to extract automatically fuzzy If-Then rules from the trained neural network. To prove the effectiveness and validity of the proposed fuzzy neural expert system, a fuzzy neural expert system for medical diagnosis has been developed. >

BACKGROUND: Currently, artificial intelligence in the form of artificial neural networks is being actively implemented in a number of areas of our lives, including medicine. In particular, in otorhinolaryngology, artificial neural networks are used to analyze images obtained during endoscopic examinations of patients (e.g., videolaryngoscopy) [1–3]. The interpretation of laryngoscopic images often presents significant difficulties for practicing physicians, which reduces the frequency of detection of precancerous laryngeal diseases and contributes to the increase in the number of patients with stage III–IV laryngeal cancer [4, 5]. This underscores the significance of prompt performance and accurate interpretation of the findings of endoscopic examinations of patients with laryngeal disorders. Artificial neural networks can be employed to analyze the results of videolaryngoscopy, furnishing the physician with supplementary information that can enhance diagnostic accuracy and diminish the probability of error [6, 7]. AIM: The study aims to develop and train an artificial neural network for recognizing characteristic features of laryngeal neoplasms and variants of laryngeal normality. MATERIALS AND METHODS: The study was conducted under the grant of the Moscow Center for Innovative Technologies in Healthcare (grant No. 2112-1/22) entitled “Using Neural Networks (Artificial Intelligence Algorithms) for Control and Improving the Quality of Diagnosis and Treatment of Diseases of Laryngeal and Ear Structures through Digital Technologies”.The following methods were used during the course of the study: data collection for the creation of a photobank (dataset) of medical images obtained during videolaryngoscopy; data partitioning for the formation of datasets for individual nosologies and groups of diseases; the method of consilium; analysis of the accuracy of recognition and classification of digital endoscopic images; and training of classification neural networks. Consequently, a dataset comprising 1,471 laryngeal images in digital formats (JPEG, BMP) was assembled, labelled, and uploaded for the purpose of training the artificial neural network. Of the total number of images, 410 were classified as pertaining to laryngeal formation, while 1061 were classified as variants of normality. Subsequently, the neural network was trained and tested to identify the signs of normal and laryngeal masses. RESULTS: The results of the testing of the artificial neural network indicated the formation of an inaccuracy matrix, the calculation of the value of recognition accuracy, the calculation of the quality indicators of the model performance, and the construction of the ROC curve. The developed and trained artificial neural network demonstrated an accuracy of 86% in recognizing the signs of laryngeal masses and norms. CONCLUSIONS: This study demonstrates that a trained artificial neural network can successfully distinguish between signs of normal and laryngeal masses in endoscopic photographs. With further training of the neural network and achievement of high accuracy, this technology can be used in clinical practice as an assistant in the interpretation of laryngoscopic images and early diagnosis of laryngeal masses. It can also be employed to control and improve the quality of diagnosis and treatment of diseases of the throat, nose, and ears by primary care physicians.

Training of neural network can be accelerated by limited numerical precision together with specialized low-precision hardware. This paper studies how low precision can impact on entire training of RNNs. We emulate low precision training for recently proposed gated recurrent unit (GRU) and use dynamic fixed point as a target numeric format. We first show that batch normalization on input sequences can help speed up training with low precision as well as high precision. We also show that the overflow rate should be carefully controlled for dynamic fixed point. We study low precision training with various rounding options including bit truncation, round to nearest, and stochastic rounding. Stochastic rounding shows superior results than the other options. The effect of fully low precision training is also analyzed by comparing partial low precision training. We show that the piecewise linear activation function with stochastic rounding can achieve comparable training results with floating point precision. Low precision multiplier and accumulator (MAC) with linear-feedback shift register (LFSR) is implemented with 28nm Synopsys PDK for energy and performance analysis. Implementation results show low precision hardware is 4.7x faster, and energy per task is up to 4.55x lower than that of floating point hardware.

Traditionally, the application of a neural network (Haykin, 1999) to solve a problem has required to follow some steps before to obtain the desired network. Some of these steps are the data preprocessing, model selection, topology optimization and then the training. It is usual to spend a large amount of computational time and human interaction to perform each task of before and, particularly, in the topology optimization and network training. There have been many proposals to reduce the effort necessary to do these tasks and to provide the experts with a robust methodology. For example, Giles et al. (1995) provides a constructive method to optimize iteratively the topology of a recurrent network. Other methods attempt to reduce the complexity of the network structure by mean of removing unnecessary network nodes and connections like in (Morse, 1994). In the last years, evolutionary algorithms have been shown as promising tools to solve this problem, existing many competitive approaches in the literature. For example, Blanco et al. (2001) proposed a master-slave genetic algorithm to train (master algorithm) and to optimize the size of the network (slave algorithm). For a general view of the problem and the use of evolutionary algorithms for neural network training and optimization, we refer the reader to (Yao, 1999). Although the literature about genetic algorithms and neural networks is very extensive, we would like to remark the recent popularity of multi-objective optimization (Coello et al., 2002, Jin, 2006), specially to solve the problem of simultaneous training and topology optimization of neural networks. These methods have shown to perform suitably for this task in previous works, although most of them are proposed for feedforward models. They attempt to optimize the structure of the network (number of connections, hidden units or layers), while training the network at the same time. Multi-objective algorithms may provide important advantages in the simultaneous training and optimization of neural networks: They may force the search to return a set of optimal networks instead of a single one; they are capable to speed-up the optimization process; they may be preferred to a weight-aggregation procedure to cover the regularization problem in neural networks; and they are more suitable when the designer would like to combine different error measures for the training. A recent review of these techniques may be found in (Jin, 2006).

ABSTRACTA back‐propagation neural network was applied to predicting the KIC values using tensile material data and investigating the effects of crack plane orientation and temperature. The 595 KIC data of structural steels were used for training and testing the neural network model. In the trained neural network model, yield stress has relatively the most effect on KIC value among tensile material properties and KIC value was more sensitive to KIC test temperature than to crack plane orientation valid in the range of material data covered in this study. The performance of the trained artificial neural network (ANN) was evaluated by comparing output of the ANN with results of a conventional least squares fit to an assumed shape. The conventional linear or nonlinear least squares fitting methods gave very poor fitting results but the results predicted by the trained neural network were considerably satisfactory. This study shows that the neural network can be a good tool to predict KIC values according to the variation of the temperature and the crack plane orientation using tensile test results.

This paper focuses on one half of the knowledge acquisition problem for fuzzy systems, namely the acquisition of a fuzzy rule base from a set of input/output data, and in particular on the extraction of a set of fuzzy rules from a trained neural network. Some limitations with previously reported work in this area are first identified. Two simple rule extraction techniques are then described and tested on a well known classification problem. The performance of the resultant rule bases compares more favourably than those reported using alternative techniques.

In this paper, we present a new key generation model for image hashing using neural network, which does not embed any data into the content but is able to extract meaningful data from target image. This model trains artificial neural network to assign predefined code and uses this trained artificial neural network weight and the coordinates of the selected feature sub blocks of target image as keys to extract the predefined code. In this model, the observed output signal from the trained neural network is used as image hash value which distinguishes the target image from other images. The proposed method contributes to secure image hashing for content identification without damaging or losing any detailed data of visual images. The proposed method realizes an application for image authentication, image similarity comparison, verification of image integrity and copyright protection of multimedia contents.

We report the development and implementation of electronic excited-state capabilities for semiempirical quantum chemical methods at both the Configuration Interaction Singles and Time-Dependent Hartree-Fock levels of theory, integrated within the PYSEQM 2.0 software package (https://github.com/lanl/PYSEQM). PYSEQM is a Python-based package designed for efficient and scalable quantum chemical simulations. Leveraging the PyTorch framework enables PYSEQM to benefit from automatic differentiation and GPU acceleration, leading to substantial performance gains in molecular property evaluations. In particular, our implementation enables efficient calculation of excited-state properties for large molecular systems. Benchmarking on systems with up to a thousand atoms demonstrates that excited-state computations can be completed in under a minute on modern GPUs, making this approach particularly suitable for high-throughput screening, real-time feedback in interactive simulations, and large-scale dynamical studies. Additionally, PYSEQM includes a machine learning interface that supports Hamiltonian parameter reoptimization and neural network training. These capabilities open new avenues for data-driven excited-state dynamics simulations, offering a path toward combining quantum chemical rigor with machine learning efficiency. Overall, this work facilitates access to excited-state quantum chemistry for large systems, while laying the foundation for future hybrid quantum-machine-learning approaches in photochemistry, photophysics, and materials discovery.

On the training and performance of high-order neural networks

Accurate and robust risk prediction methods are of critical importance in calculating insurance costs. In the present paper, we study the case of vehicle insurance and develop a computational intelligence based method for obtaining risk estimates based on the data provided by the client to the insurance company. The method is based on analyzing the contracts, processing the input data, applying classification, encoding the processed data in a way that it can be used as input to an artificial neural network, and finally, the training of the neural networks. As a result, an application computing a risk estimate for a particular contract is developed. In the paper we discuss issues related to processing of the data and training of neural networks, and analyze the obtained results.