Long Short-Term Sample Distillation

<p>The article discusses the use of artificial neural networks in financial markets, due to the fact that this direction is very promising. Neural networks are well suited for tasks in which there are a large number of influencing factors, both known and unknown. The existing methods of neural networks training are considered, their pros and cons are noted. The use of neural networks in forecasting financial time series in real time is faced with the problem of the duration of the learning process of the neural network and the selection of significant inputs of the neural network. This problem can be solved by using the high-speed method of teaching the perceptron. The high-speed method of training allows for a much smaller number of iterations to train a multilayer perceptron on a given set of examples compared to the method of back propagation of the error. The high-speed method of training a multilayer perceptron allows us to assess whether it is possible to reach a given learning error of the neural network or not. This method of training is proposed to be used as an evaluation in the case of critical tasks before applying the method of back propagation of the error, in the case of non-critical tasks, the results of the neural network trained by this method can be used directly.</p>

ScieNet: Deep learning with spike-assisted contextual information extraction

Supervised Neural Network (NN) learning is a process in which input patterns and known targets are presented to a NN while it learns to recognize (classify, map, fit, etc.) them as desired. The learning is mathematically defined as an optimisation problem, i.e., an error function representing the differences between the desired and actual output, is being minimized (Bishop, 1995; Haykin, 1999). Because the most popular supervised learning techniques are gradient based (Backpropagation BP), they suffer from the so-called Local Minima Problem (Bishop, 1995). This has motivated the employment of Global Optimisation (GO) methods for the supervised NN learning. Stochastic and heuristic GO approaches including Evolutionary Algorithms (EA) demonstrated promising performance over the last decades (Smagt, 1994; Sexton et al., 1998; Jordanov & Georgieva, 2007; etc.). EA appeared more powerful than BP and its modifications (Sexton et al., 1998; Alba & Chicone 2004), but hybrid methods that combine the advantages of one or more GO techniques and local searches were proven to be even better (Yao, 1999; Rocha et al., 2003; Alba & Chicano, 2004; Ludemir et al., 2006). Hybrid methods were promoted over local searches and simple population based techniques in Alba & Chicone (2004). The authors compared five methods: two BP implementations (gradient descent and Levenberg-Marquardt), Genetic Algorithms (GA), and two hybrid methods, combining GA with different local methods. The methods were used for NN learning applied to problems arising in medicine. Ludemir et al. (2006) optimized simultaneously NN weights and topology with a hybrid method combining Simulated Annealing (SA), Tabu Search (TS) and BP. A set of new solutions was generated on each iteration by TS rules, but the best solution was only accepted according to the probability distribution as in conventional SA. Meanwhile, the topology of the NN was also optimized and the best solution was kept. Finally, BP was used to train the best NN topology found in the previous stages. The new methodology compared favorably with SA and TS on four classification and one prediction problems. Plaginakos et al. (2001) performed several experiments to evaluate various training methods – six Differential Evolution (DE) implementations (with different mutation operators), BP, BPD (BP with deflection), SA, hybridization of BP and SA (BPSA), and GA. They reported O pe n A cc es s D at ab as e w w w .in te ch w eb .o rg

This dissertation introduces a new system for handwritten text recognition based on an improved neural network design. Most of the existing neural networks treat mean square error function as the standard error function. The system as proposed in this dissertation utilizes the mean quartic error function, where the third and fourth derivatives are non-zero. Consequently, many improvements on the training methods were achieved. The training results are carefully assessed before and after the update. To evaluate the performance of a training system, there are three essential factors to be considered, and they are from high to low importance priority: 1) error rate on testing set, 2) processing time needed to recognize a segmented character and 3) the total training time and subsequently the total testing time. It is observed that bounded training methods accelerate the training process, while semi-third order training methods, next-minimal training methods, and preprocessing operations reduce the error rate on the testing set. Empirical observations suggest that two combinations of training methods are needed for different case character recognition. Since character segmentation is required for word and sentence recognition, this dissertation provides also an effective rule-based segmentation method, which is different from the conventional adaptive segmentation methods. Dictionary-based correction is utilized to correct mistakes resulting from the recognition and segmentation phases. The integration of the segmentation methods with the handwritten character recognition algorithm yielded an accuracy of 92% for lower case characters and 97% for upper case characters. In the testing phase, the database consists of 20,000 handwritten characters, with 10,000 for each case. The testing phase on the recognition 10,000 handwritten characters required 8.5 seconds in processing time.

Event Abstract Back to Event Logarithmic Hybrid Optical Neural Network-type systems: towards digital-optical cognitive models of robust object recognition Ioannis Kypraios1* 1 Oxford University, Centre for Innovation & Enterprise, APEM Computing Lab, United Kingdom Abstract - By combining the complex logarithmic r-theta mapping of a space-variant imaging sensor [1, 2] with the hybrid digital-optical neural network filter together with a window unit multiple objects of the same class or of different classes can robustly be recognized. The resulted logarithmic r-theta mapping for hybrid digital-optical neural network system or briefly referred to as logarithmic hybrid optical neural network system is shown to exhibit with a single pass over the input data simultaneously in-plane rotation, out-of-plane rotation, scale, log r-θ map translation and shift invariance, and good clutter tolerance by recognizing correctly the different objects within the cluttered scenes. Here, we study the biologically-inspired knowledge learning and knowledge representation [3] achieved by the logarithmic hybrid optical neural network-type of object recognition systems (see Fig. 1 and Fig. 2). We investigate the effects that altering the knowledge representation can have on the problem’s learned knowledge and the problem solving process, in overall. Further, we study the logarithmic unconstrained-, logarithmic constrained-, and logarithmic modified-hybrid optical neural network systems architectures’ designs [4, 5]. We show how the logarithmic unconstrained-hybrid optical neural network object recognition system applies an unconstrained representation of the problem’s knowledge to maximize the search of solutions, how the logarithmic constrained-hybrid optical neural network object recognition system uses a constrained representation of the problem’s knowledge to guide the search towards certain solutions over others in the multidimensional search space, and how the logarithmic modified-hybrid optical neural network object recognition system uses memory-like masks to recall certain solutions over others in the multidimensional search space. Fig. 1. (a) Simplified human retina and visual cortex model used for description purposes only; (b) A digital-optical computational model for the cognitive interaction between the retina and the human visual cortex with the general logarithmic hybrid optical neural network architercture. Fig. 2. Biologically-inspired knowledge learning and knowledge representation with the logarithmic hybrid optical neural network-type of object recognition systems. Figure 1 Figure 2 References [1] C. F. R. Weiman, “Video compression via log-polar mapping”, Real-Time Image Processing II, SPIE Symposium on OE/Aerospace Sensing, Vol. 1295, 266-277 (1990) [2] C. G. Ho et al., “Sensor geometry and sampling methods for space-variant image processing”, Pattern Analysis and Applications, Vol. 5, 369-384 (2002) [3] I. Lee, and B. Portier, "An empirical study of knowledge representation and learning within conceptual spaces for intelligent agents", 6th IEEE/ACIS International Conference Computer and Information Science, 463-468, Melbourne, Australia (2007) [4] I. Kypraios et al., “Fully invariant complex Logarithmic r- map for the Hybrid Optical Neural Network filter for object recognition within cluttered scenes”, 50th Anniversary International Symposium ELMAR 2008, IEEE Region 8/IEEE Croatia Section/EURASIP, Zadar-Croatia, Vol. 1, 141-146 (2008) [5] I. Kypraios et al., “Logarithmic r-θ mapping for hybrid optical neural network filter for object recogntion within cluttered scenes”, Recent Advances in Multimedia Signal Processing and Communications, SCI231, Springer-Verlag Berlin Heidelberg, 91-120 (2009) Keywords: digital-optical, cognitive, object recognition, Robust and Adaptive Systems, Knowledge representation, learning and memory Conference: Neuroinformatics 2013, Stockholm, Sweden, 27 Aug - 29 Aug, 2013. Presentation Type: Poster Topic: Neuromorphic engineering Citation: Kypraios I (2013). Logarithmic Hybrid Optical Neural Network-type systems: towards digital-optical cognitive models of robust object recognition. Front. Neuroinform. Conference Abstract: Neuroinformatics 2013. doi: 10.3389/conf.fninf.2013.09.00066 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 09 Apr 2013; Published Online: 11 Jul 2013. * Correspondence: Dr. Ioannis Kypraios, Oxford University, Centre for Innovation & Enterprise, APEM Computing Lab, Begbroke, Oxfordshire, OX5 1PF, United Kingdom, ioanniskyp@yahoo.com Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Ioannis Kypraios Google Ioannis Kypraios Google Scholar Ioannis Kypraios PubMed Ioannis Kypraios Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Deep Neural Networks (DNNs) have been successfully applied to various real-world machine learning applications. However, performing large DNN inference tasks in real-time remains a challenge due to its substantial computational costs. Recently, Spiking Neural Networks (SNNs) have emerged as an alternative way of processing DNN'fs task. Due to its eventbased, data-driven computation, SNN reduces both inference latency and complexity. With efficient conversion methods from traditional DNN, SNN exhibits similar accuracy, while leveraging many state-of-the-art network models and training methods. In this work, an efficient neuromorphic hardware architecture for image recognition task is presented. To preserve accuracy, the analog-to-spiking conversion algorithm is adopted. The system aims to minimize hardware area cost and power consumption, enabling neuromorphic hardware processing in edge devices. Simulation results have shown that, with the MNIST digit recognition task, the system has achieved $\times 20$ reduction in terms of core area cost compared to the state-of-the-art works, with an accuracy of 94.4%, core area of 15 $\mu m^{2}$ at a maximum frequency of 250 MHz.

This paper proposes a new training method for the phoneme identification neural network called “neural fuzzy training.” In the proposed training, nondeterministic (fuzzy) class information is assigned to the training signal, in contrast to the traditional method where a deterministic class information is assigned.This study aims at the realization of a robust neural network, thereby improving the cumulative recognition rate of the phoneme identification and avoiding overtraining. The proposed neural fuzzy training is realized by backpropagation. In the conventional training, a deterministic phoneme class information is assigned to the training signal of the neural network as the value 1 or 0. However, in the proposed training, the fuzzy class information is assigned to the training signal for each training sample as the likelihood value between 0 and 1.In the proposed training method, the likelihood is calculated by the monotonically decreasing function (such as exp(−α · d2)) of the distance between the training sample and the closest sample belonging to each phoneme class. The proposed neural fuzzy training method has a problem in that a large amount of computation cost is required since the training signal is determined by calculating the distances to all training samples. To solve this problem, the representative samples in each phoneme class are defined and the likelihood to the phoneme classes are determined by calculating the distance between the representative sample and the training sample.By this simplification of the likelihood calculation, the computational cost to determine the training signal is reduced considerably. To demonstrate the usefulness of the neural fuzzy training, an experiment is conducted: /b, d, g, m, n, N/ identification, 18 consonant identification and phrase recognition using TDNN‐LR. The ATR database is used in the experiment. In the phoneme identification experiment, the speech samples which are extracted using the hand‐label is used. The TDNN is trained using speed samples uttered in word style, and the evaluation is performed using speech samples uttered in phrase style and in sentence style.In the phrase recognition experiment using TDNN‐LR, the TDNN is trained using speed samples uttered word style using a hand label. The evaluation is performed using speech samples uttered in phrase style. In either experiment, an improvement of using the fuzzy training can be observed. Especially, in the phrase recognition experiment using TDNN‐LR, the top recognition rate is improved from 71.2 percent to 80.9 percent, and the top 5th recognition rate is improved from 92.8 percent to 96.O percent. Furthermore, it appeared also that the neural fuzzy training is a high‐speed training method.

Neural network (NN) based prediction or detection systems often perform excellently with easy problems without considering 1-5% difficult problems. This work proposes an adversarial NN training method for constructing the prediction interval (PI). The proposed training method considers adverse situations where traditional NN based PIs frequently fail. First, the conventional lower upper bound estimation (LUBE) method is applied in parallel for initial training of NNs with different initialization. Each NN based PI fails to cover a few samples. Input combinations of those samples are adversely changed by a small amount to generate the adverse samples. A new dataset is generated by appending adverse samples. Finally, an NN is trained with the adverse dataset. The method is applied to construct the NN for wind power prediction. According to the result analysis, the proposed method performs better in adverse situations.

Introduction. The article considers the solution of the problem of detecting fires at technological facilities in automatic mode. To solve this problem, an approach was chosen to create a convolutional neural network capable of operating on a real-time video stream. Aims and Purposes. The aim of this work is to create a neural network capable of detecting flames and smoke in the image from CCTV cameras. The purposes of the work include: selection of the optimal architecture of the neural network in accordance with the latest research in this field; speeding up the chosen architecture using quantization and filter thinning techniques. Methods. Different architectures of convolutional neural networks performing the task of detecting objects in an image are considered. Their performance and quality of work are compared. The YOLOv5 architecture, its target function, training methods and ways of speeding up work are considered. Results and discu ssion. The paper shows the results of training a convolutional neural network of the YOLOv5 architecture for the task of flame and smoke detection, as well as how the results change when applying neural network acceleration methods. It was determined that the use of such acceleration methods such as quantization and filter cleaning can significantly increase up the speed of the neural network, while almost no loss in accuracy of operation. Conclusions. As a result of the conducted work, the architecture of the neural network was determined for performing the task of detecting a fire source. Based on the chosen architecture, a neural network was trained to detect flames and smoke in the image. The speed of its work allows to process video stream in real-time without using graphic accelerator.

The field of neural network training and optimization has seen significant advancements in recent years, with new techniques and algorithms being proposed to improve the efficiency and effectiveness of training. In this paper, we review several key optimization techniques and their impact on training neural networks, with a focus on long-term dependencies and the difficulties that can arise during training. We begin by discussing the challenges of learning long-term dependencies with gradient descent, as highlighted in the 1994 paper by Bengio et al. We then introduce Adam, a method for stochastic optimization proposed by Kingma and Ba in 2014. We also explore the difficulties of training recurrent neural networks, as discussed in the 2013 paper by Pascanu, Mikolov, and Bengio. We also review recent advances in optimization techniques such as Convergence of Adam and Beyond by Jianmin et al. (2017), Yogi by Dong et al. (2018), AdaBound by Zhang et al. (2018) and On the Variance of the Adaptive Learning Rate and Beyond by Liu et al. (2019). We will highlight the advantages and disadvantages of each technique and discuss their potential impact on the field. Overall, this paper provides a comprehensive overview of recent advancements in neural network optimization and their implications for training and performance. Keywords: Deep neural networks, optimization, Adam, gradient clipping, training, stabilization, overfitting, generalization, recurrent neural networks, non-convex loss landscapes.

Advanced Driver Assistance System (ADAS) is a very important part of an up to date vehicle. For achieving highlevel objectives in such ADAS functionality like LKA (lane keeping assistance), LDW (lane departure warning system), FCW (forward collision warning) the quality of the algorithms under the hood must be extremely high. In the last few years, it is common that these algorithms are based on DNNs (deep neural networks) applied to the tasks of semantic and instance segmentation, 2D/3D object detection and visual object tracking. Recent state-of-the-art DNN models as usual solve only one single task from the listed above and running several neural networks is rather computationally expensive and even impossible due to the lack of the GPU memory. One of the approaches used to overcome such a problem is a shared backbone (also called feature extractor or encoder). The backbone consumes most of the computing resources thus the model with a shared backbone achieves better inference performance. Unfortunately, the training procedure for a shared backbone model has several difficulties. The first one is the lack of datasets with all the required and uniform annotation types. The second problem is a more sophisticated backpropagation procedure. In this paper, we consider several methods for multi-task neural network training and present the results of such training procedures on several public datasets with dissimilar annotation types. The shared backbone is applied to the following three tasks performed simultaneously on the road scene: semantic segmentation, 2D object detection and 3D object detection. While the performance of the DNNs with shared backbone increased significantly, we obtained the quality evaluation results, which are quite close to the original separate state-of-the-art DNNs and even outperforms them in some evaluation indices.

A multilayer neural network has been designed to forecast average daily energy consumption of a railway marshalling yard. The suggested model comprises a multilayer perceptron using 22 inputs, the n-th number of hidden layers and one output. The number of hidden layers in the neural network and neurons in them was chosen experimentally. A comparative selection of activation functions and training methods has allowed for all other parameters to achieve a minimum average relative error. Two types of loads corresponding to holidays (non-working) and working days were identified. An additional input node with binary coding and two nodes for coding the season were introduced due to a certain repeatability characterizing samples of prediction of loads of energy consumption of the marshalling yard depending on type of a day and on a season. As accounting of the dependence of the forecast on load values in previous days and years (dynamic dependencies) is most important factor, this neural network takes into account the average daily energy consumption during four days of the current period, precedingthe forecasted date, and the average daily power consumption during four days prior to this date during last three years.As a result, considering all factors and experimentally selected parameters of the neural network, the minimum resulting error of MAPE is about 1,4 %, which shows the advantage of the developed neural network in comparison with two other methods of solution of the problem, suggested by other researchers.

This article is devoted to the solution of the problem of constructing a neuro-fuzzy classifier for assessing the level of students' knowledge. Currently, intelligent systems for assessing complex technical, biophysical and socio-economic systems have undergone rapid development and are often based on the use of neural networks, fuzzy logic methods and procedures for forming rule bases based on data analysis. In this work, a neuro-fuzzy classifier was used to assess the level of students' knowledge. The User Knowledge Modeling dataset for training the fuzzy neural network was obtained from the open source Machine Learning Repository. The number of copies in the dataset is 403, and the number of attributes for classification is 5. Four gradations are distinguished for the level of students' knowledge - very low, low, medium and high. This corresponds to the Russian assessment system: unsatisfactory, satisfactory, good and excellent. In the Deductor environment, a preliminary analysis of data quality, work with outliers and extreme values, checking for duplicates and contradictions, as well as correlation analysis of input and output parameters were carried out. After processing, the data were randomly divided into training and test samples in the ratio of 70% and 30%, respectively. To build a neuro-fuzzy model, the ANFIS editor from the MATLAB application package was used. The fuzzy inference system contained 5 input variables with 3 fuzzy gradations and 1 output variable. As a result of training, 128 fuzzy rules corresponding to the model structure were formed. When building a neuro-fuzzy classifier, fuzzy neural networks with various combinations of the training method, type of membership function of the input and output parameters were used. The lowest value of the root mean square error, equal to 0.16, was achieved in the model with the following parameters: training method - back propagation of the error, membership function of the input variables - double Gaussian, membership function of the output variable - constant. To assess the adequacy of the neuro-fuzzy model, errors of the first and second kind were calculated. The error of the first kind on the test data was 0%, the error of the second kind - 4.6%. The accuracy of the neuro-fuzzy model was 95.6% and was higher than when using other methods of data mining: neural network, decision tree, linear and logistic regression. Thus, the constructed neuro-fuzzy model showed high accuracy in assessing students' knowledge and can be effectively used in the educational process to solve the task.

In a previous paper, we introduced a deep learning neural network that should be able to detect the existence of very shallow periodic planetary transits in the presence of red noise. The network in that feasibility study would not provide any further details about the detected transits. The current paper completes this missing part. We present a neural network that tags samples that were obtained during transits. This is essentially similar to the task of identifying the semantic context of each pixel in an image—an important task in computer vision, called “semantic segmentation,” which is often performed by deep neural networks. The neural network we present makes use of novel deep learning concepts such as U-Nets, Generative Adversarial Networks, and adversarial loss. The resulting segmentation should allow further studies of the light curves that are tagged as containing transits. This approach toward the detection and study of very shallow transits is bound to play a significant role in future space-based transit surveys such as PLATO, which are specifically aimed to detect those extremely difficult cases of long-period shallow transits. Our segmentation network also adds to the growing toolbox of deep learning approaches that are being increasingly used in the study of exoplanets; but, so far mainly for vetting transits, rather than their initial detection.

Convolutional neural networks have powerful performances in many visual tasks because of their hierarchical structures and powerful feature extraction capabilities. SPD (symmetric positive definition) matrix is paid attention to in visual classification, because it has excellent ability to learn proper statistical representation and distinguish samples with different information. In this paper, a deep neural network signal detection method based on spectral convolution features is proposed. In this method, local features extracted from convolutional neural network are used to construct the SPD matrix, and a deep learning algorithm for the SPD matrix is used to detect target signals. Feature maps extracted by two kinds of convolutional neural network models are applied in this study. Based on this method, signal detection has become a binary classification problem of signals in samples. In order to prove the availability and superiority of this method, simulated and semi-physical simulated data sets are used. The results show that, under low SCR (signal-to-clutter ratio), compared with the spectral signal detection method based on the deep neural network, this method can obtain a gain of 0.5–2 dB on simulated data sets and semi-physical simulated data sets.