Optical Recognition of Handwritten Logic Formulas Using Neural Networks

Stochastic differential equation (SDE) models are the foundation for pricing and hedging financial derivatives. The drift and volatility functions in SDE models are typically chosen to be algebraic functions with a small number ( < 5 ) of parameters which can be calibrated to market data. A more flexible approach is to use neural networks to model the drift and volatility functions, which provides more degrees-of-freedom to match observed market data. Training of models requires optimizing over an SDE, which is computationally challenging. For European options, we develop a fast stochastic gradient descent (SGD) algorithm for training the neural network-SDE model. Our SGD algorithm uses two independent SDE paths to obtain an unbiased estimate of the direction of steepest descent. For American options, we optimize over the corresponding Kolmogorov partial differential equation (PDE). The neural network appears as coefficient functions in the PDE. Models are trained on large datasets (many contracts), requiring either large simulations (many Monte Carlo samples for the stock price paths) or large numbers of PDEs (a PDE must be solved for each contract). Numerical results are presented for real market data including S&P 500 index options, S&P 100 index options, and single-stock American options. The neural-network-based SDE models are compared against the Black-Scholes model, the Dupire's local volatility model, and the Heston model. Models are evaluated in terms of how accurate they are at pricing out-of-sample financial derivatives, which is a core task in derivative pricing at financial institutions. Specifically, we calibrate a neural network-SDE model to market data for a financial derivative on an asset with price S t with a payoff function g ( s ) , and we then evaluate its generalization accuracy for a financial derivative on the same asset S t but with a different payoff function f ( s ) . In addition to comparing out-of-sample pricing accuracy, we evaluate the hedging performance of the neural network-SDE model.

Speed and accuracy investigations of neural network algorithms for ionospheric modelling at an equatorial region

Due to increased usage of digital technologies in all sectors and in almost all day to day activities to store and pass information, Handwriting character recognition has become a popular subject of research. Handwriting remains relevant, but people still want to have Handwriting copies converted into electronic copies that can be communicated and stored electronically. Handwriting character recognition refers to the computer's ability to detect and interpret intelligible Handwriting input from Handwriting sources such as touch screens, photographs, paper documents, and other sources. Handwriting characters remain complex since different individuals have different handwriting styles. This paper aims to report the development of a Handwriting character recognition system that will be used to read students and lectures Handwriting notes. The development is based on an artificial neural network, which is a field of study in artificial intelligence. Different techniques and methods are used to develop a Handwriting character recognition system. However, few of them focus on neural networks. The use of neural networks for recognizing Handwriting characters is more efficient and robust compared with other computing techniques. The paper also outlines the methodology, design, and architecture of the Handwriting character recognition system and testing and results of the system development. The aim is to demonstrate the effectiveness of neural networks for Handwriting character recognition.

We consider in this work a system of two stochastic differential equations named the perturbed compositional gradient flow. By introducing a separation of fast and slow scales of the two equations, we show that the limit of the slow motion is given by an averaged ordinary differential equation. We then demonstrate that the deviation of the slow motion from the averaged equation, after proper rescaling, converges to a stochastic process with Gaussian inputs. This indicates that the slow motion can be approximated in the weak sense by a standard perturbed gradient flow or the continuous-time stochastic gradient descent algorithm that solves the optimization problem for a composition of two functions. As an application, the perturbed compositional gradient flow corresponds to the diffusion limit of the Stochastic Composite Gradient Descent (SCGD) algorithm for minimizing a composition of two expected-value functions in the optimization literatures. For the strongly convex case, such an analysis implies that the SCGD algorithm has the same convergence time asymptotic as the classical stochastic gradient descent algorithm. Thus it validates, at the level of continuous approximation, the effectiveness of using the SCGD algorithm in the strongly convex case.

Linear prediction methods, such as least squares for regression, logistic regression and support vector machines for classification, have been extensively used in statistics and machine learning. In this paper, we study stochastic gradient descent (SGD) algorithms on regularized forms of linear prediction methods. This class of methods, related to online algorithms such as perceptron, are both efficient and very simple to implement. We obtain numerical rate of convergence for such algorithms, and discuss its implications. Experiments on text data will be provided to demonstrate numerical and statistical consequences of our theoretical findings.

In the paper, we consider a stochastic gradient descent algorithm in combination with a dropout method. We used the theory of intuitionistic fuzzy sets for the assessment of the equivalence of the respective assessment units. We also consider a degree of uncertainty when the information is not enough.

Structural health monitoring (SHM) provides an economic approach which aims to enhance understanding the behavior of structures by continuously collecting data through multiple networked sensors attached to the structure. These data are then utilized to gain insight into the health of a structure and make timely and economic decisions about its maintenance. The generated SHM sensing data are non-stationary and exist in a correlated multi-way form which makes the batch/off-line learning and standard two-way matrix analysis unable to capture all of these correlations and relationships. In this sense, the online tensor data analysis has become an essential tool for capturing underlying structures in higher-order datasets stored in a tensor $${\mathcal {X}} \in {\mathbb {R}} ^{I_1 \times \cdots \times I_N} $$ X ∈ R I 1 × ⋯ × I N . The CANDECOMP/PARAFAC (CP) decomposition has been extensively studied and applied to approximate X by N loading matrices $$A(1),\ldots ,A(N)$$ A ( 1 ) , … , A ( N ) where N represents the order of the tensor. We propose a novel algorithm, FP-CPD, to parallelize the CANDECOMP/PARAFAC (CP) decomposition of a tensor $${\mathcal {X}} \in {\mathbb {R}} ^{I_1 \times \cdots \times I_N} $$ X ∈ R I 1 × ⋯ × I N . Our approach is based on stochastic gradient descent (SGD) algorithm which allows us to parallelize the learning process, and it is very useful in online setting since it updates $${\mathcal {X}}^{t+1}$$ X t + 1 in one single step. Our SGD algorithm is augmented with Nesterov’s accelerated gradient and perturbation methods to accelerate and guarantee convergence. The experimental results using laboratory-based and real-life structural datasets indicate fast convergence and good scalability.

Artificial neural network (ANN) models have been used in the past to model surface roughness in manufacturing processes. Specifically, different parameters influence surface roughness in fused filament fabrication (FFF) processes. In addition, the characteristics of the networks have a direct impact on the performance of the models. In this work, a study about the use of ANN to model surface roughness in FFF processes is presented. The main objective of the paper is discovering how key ANN parameters (specifically, the number of neurons, the training algorithm, and the percentage of training and validation datasets) affect the accuracy of surface roughness predictions. To address this question, 125 3D printing experiments were conducted changing orientation angle, layer height and printing temperature, and measuring average roughness Ra as response. A multilayer perceptron neural network model with backpropagation algorithm was used. The study evaluates the effect of three ANN parameters: (1) number of neurons in the hidden layer (4, 5, 6 or 7), (2) training algorithm (Levenberg–Marquardt, Resilient Backpropagation or Scaled Conjugate Gradient), and (3) data splitting ratios (70%–15%–15% vs. 55%–15%–30%). Mean Absolute Error (MAE) was used as the performance metric. The Resilient Backpropagation algorithm, 7 neurons, and using 55% of training data yielded the best predictive performance, minimizing the MAE. Additionally, the impact of the dataset size on prediction accuracy was analysed. It was observed that the performance of the ANN gets worse as the number of datasets is reduced, emphasizing the importance of having sufficient data. This study will help to select appropriate values for the printing parameters in FFF processes, as well as to define the characteristics of the ANN to be used to model surface roughness.

In feature learning field, many methods are inspired by advances in neuroscience. Among them, neural network and sparse coding have been broadly studied. Predictive sparse decomposition (PSD) is a practical variant of these two methods. It trains a neural network to estimate the sparse codes. After training, the neural network is fine-tuned to achieve higher performance on object recognition tasks. It is widely believed that introducing discriminative information can make the features more useful for classification task. Hence, in this work, we propose applying the task-driven dictionary learning framework to the PSD and demonstrate that this new model can be optimized by the stochastic gradient descent (SGD) algorithm. Before our work, the semi-supervised auto-encoder framework has already been proposed to guide neural network to extract discriminative representations. But it does not improve the classification performance of neural network. In the experiments, we compare the proposed method with the semi-supervised auto-encoder method. The performance of PSD is used as the baseline for these two methods. On the MNIST and USPS datasets, our method can generate more discriminative and predictable sparse codes than other methods. Furthermore, the recognition accuracy of neural network can be improved.

Adaptive training is critical for image classification by artificial neural networks (ANNS). While the machine learning community has been enthusiastic in developing various training algorithms, little research has been conducted to evaluate the performance of these algorithms in image classification by neural networks. We introduce and evaluate nine commonly-used training algorithms in terms of their performance in land-cover classification from remotely sensed data by the multi-layer-perceptron (MLP) neural networks. ALP has been considered as the most popular neural network architecture. The training algorithms we consider are Steepest Gradient Descent, Gradient Descent with Momentum, Resilient Propagation, Fletcher-Reeves, Polak-Ribiere, Powell-Beale, Scaled Conjugate Gradient, BFGS (Broyden, Fletcher, Goldfarb, and Shanno), and Levenberg-Marquardt. We use each algorithm to train the MLP networks multiple times using identical training samples, and then apply each of the resultant network models to derive land-cover information from a Landsat Enhanced Thematic Mapper Plus (ETM+) image. The training algorithms are further evaluated according to their training efficiency, capability of convergence, classification accuracy, and stability of the classification accuracy. It is found that the performance of these algorithms varies substantially and selecting an appropriate algorithm can lead to a fast and efficient training and an increase in land-cover classification accuracy by artificial neural networks.

The Stochastic gradient descent algorithm (SGD) is a classical algorithm for model optimization in machine learning. Introducing a differential privacy model to avoid privacy leakages in the optimization iteration process can achieve the balance of training accuracy and data availability. In addition, a fixed number of iterations are chosen in a conventional implementation scheme. At each iteration, parameters are updated with a noisy gradient. However, the privacy budget is mostly split evenly to each iteration without taking into account the difference in the privacy leakage risk under optimal processing. In this paper, we improve the SGD-based algorithms by appropriately allocating the privacy budget for each iteration. Intuitively, the gradient value is inversely proportional to the number of iterations. The closer the parameter is to its optimal objective value, the smaller the gradient is, and hence the gradients need to be measured more accurately. We propose an adaptive “noise reduction” algorithm that can be applied to private SGD-based empirical risk minimization (ERM) algorithms, meets the accuracy constraint simultaneously. We apply our approach to the backpropagation (BP) neural network. In the experiment, we show and validate that the proposed noise parameter configuration method provides sufficient privacy protection and improves the accuracy of data utility.

In this study, performances comparison to discriminate five mental states of five artificial neural network (ANN) training methods were investigated. Wavelet Packet Transform (WPT) was used for feature extraction of the relevant frequency bands from raw electroencephalogram (EEG) signals. The five ANN training methods used were (a) Gradient Descent Back Propagation (b) Levenberg-Marquardt (c) Resilient Back Propagation (d) Conjugate Learning Gradient Back Propagation and (e) Gradient Descent Back Propagation with movementum.

Efficiency of large-scale learning is a hot topic in both academic and industry. The stochastic gradient descent (SGD) algorithm, and its extension mini-batch SGD, allow the model to be updated without scanning the whole data set. However, the use of approximate gradient leads to the uncertainty issue, slowing down the decreasing of objective function. Furthermore, such uncertainty may result in a high frequency of meaningless update on the model, causing a communication issue in parallel learning environment. In this work, we develop a batch-adaptive stochastic gradient descent (BA-SGD) algorithm, which can dynamically choose a proper batch size as learning proceeds. Particularly on the basis of Taylor extension and central limit theorem, it models the decrease of objective value as a Gaussian random walk game with rebound. In this game, a heuristic strategy of determining batch size is adopted to maximize the utility of each incremental sampling. By evaluation on multiple real data sets, we demonstrate that by smartly choosing the batch size, the BA-SGD not only conserves the fast convergence of SGD algorithm but also avoids too frequent model updates.

A novel post-processing system for a handwritten Chinese character recognition system based on a neural network classifier is presented. The recognition results for input character images, namely candidate characters and their confidence scores, as the observed features of the recognizer are classified into the most probable characters. The confusing character set is established by analyzing large-scale recognition experimental results, and the statistical characteristics for a recognizer are expressed as confusing character sets. 3755 character categories in the GB2312-80 character-set are clustered into several hundreds of groups through searching the transitive closure of the similarity matrix associated with the confusing characters of each character category. A group of neural networks for these category groups is established and trained to be a classifier in the post-processing to recover the unrecognized characters and adjust confidence scores of the candidate characters when a candidate sequence for each individual character image is given. The experimental results show that an average accuracy rate improvement of 5.6% and 3.8% for an online and an offline handwritten Chinese character recognition system are achieved respectively.

1. In this study, neural network (NN) and response surface (RS) models were developed to investigate the response [average daily gain (ADG) and feed efficiency (FE)] of young broiler chickens to dietary protein and lysine. For this purpose, data on their responses to dietary protein and lysine were extracted from the literature and separate NN and RS models were constructed. 2. Comparison between the NN and RS models revealed higher accuracy of prediction with the NN models compared to the RS models. In terms of R 2 values, the NN models developed for both ADG (R 2 = 0.923) and FE (R 2 = 0.904) were far superior to the RS models (R 2 for ADG = 0.511; R 2 for FE = 0.67). This suggests that the NN models can serve as an alternative option to conventional regression approaches including use of RS models. 3. Optimisation of the NN models developed for response to protein and lysine showed that diets containing 220.7 (g/kg of diet) protein and 12.85 (g/kg of diet) lysine maximise ADG, whereas maximum FE is achieved with diets containing 241.3 and 13.12 (g/kg) protein and lysine, respectively. Based on the optimisation results, optimal dietary protein and lysine concentrations for maximum FE in broiler chickens during the starting period are higher than for ADG.