On the classification capability of a dynamic threshold neural network.

The classification capability of a dynamic threshold neural network

IntroductionSimulating thermal convection in Earth and planetary interiors is crucial for various applications, including benchmarking numerical codes (e.g. [1], [2]), deriving scaling laws for heat transfer in complex flows (e.g. [3], [4]), determining mixing efficiency (e.g. [5], [6]) and the characteristic spatial wavelengths of convection (e.g. [7]). However, reaching a statistically-steady state in these simulations, even in 2D, can be computationally expensive. While choosing "close-enough" initial conditions can significantly speed simulations up, this selection process can be challenging, especially for systems with multiple controlling parameters. This work explores how machine learning can be leveraged to identify optimal initial conditions, ultimately accelerating numerical convection simulations on their path to a statistically-steady state.Convection model and neural network architectureWe have compiled a comprehensive dataset comprising 128 simulations of Rayleigh-Bénard convection within a rectangular domain of aspect ratio 4, characterized by variable viscosity and internal heating. A randomized parameter sweep was employed to vary three key factors: the ratio of internal heat generation to thermal driving forces (RaQ/Ra), the temperature-dependent viscosity contrast (FKT), and the pressure-dependent viscosity contrast (FKV). Each simulation was executed until the root mean square velocity and mean temperature attained a statistically stationary state. Subsequently, we extracted the time-averaged one-dimensional temperature profile, horizontally averaged across the domain, over the final 200 time steps of each simulation.Figure 1: Schematic of the neural network used to predict statistically-steady temperature profiles.We used a feedforward neural network (NN) to predict the 1D temperature profiles as a function of the control parameters (Fig. 1). We modified the formulation of the NN from the one we used in [8] in the following ways. First, instead of each training example consisting of the full temperature profile, we now predict temperature at a given spatial point in depth (y). This allows the network to learn and predict profiles on variable computational grids, making it more adaptable. This formulation also avoids the "wiggles" encountered when predicting high-dimensional profiles. Second, we use skip-connections. For a given hidden layer, the features from all the previous hidden layers are added to it before applying the activation function. Third, we use SELU instead of tanh for faster convergence. Fourth, we concatenate the parameters to the last hidden layer, a technique shown to improve learning in some cases  (e.g., [9]). Fifth, as a consequence of conditioning the NN on spatial points, we augment the data with copies of points from the bottom and top thermal boundary layers to ensure that each batch is likely to be more representative of relevant temperature features occurring at the outer boundaries of the domain. We split the dataset into training, cross-validation, and test sets with 96, 15, and 16 simulations, respectively. To test the extrapolation capacity of the network, we pick simulations for the test set only where at least one of the simulation parameters exceeds a certain threshold. The cross-validation dataset consists of interpolated values only and is used additionally to choose the best performing NN architecture and save network weights only when the cross-validation loss improves. ResultsWe test the NN against the following baselines: linear regression, kernel ridge regression, and nearest neighbor. As shown in Table 1, when predicting unseen profiles from interpolated cases in the cross-validation set or even the extrapolated cases (Test set), the NN predictions are the most accurate among the tested possibilities. We visualize some of the profiles predicted by these algorithms in the cross-validation set in Fig. 2. Linear regression results are left out for ease of visualization.  Algorithm                  Training set Cross-validation set Test set Linear Regression  0.0385 0.0388 0.0676 Kernel Ridge Regression  0.0148 0.0147 0.0371 Neural Network 0.0071  0.0071  0.0187 Nearest Neighbor   0.0 0.0282  0.0495 Table 1: Mean absolute error of the prediction of 1D temperature profiles of statistically-steady simulations based on different tested algorithms. The lowest error per dataset is highlighted in bold.The plots in Fig. 2 corroborate the obtained statistics about the accuracy of different methods. When the simulation parameters we are predicting happen to be close to a simulation in the training set, the nearest neighbor profile already provides a good estimate. Otherwise, the NN predictions appear to be the most accurate. This is encouraging because it shows that even for small datasets such as the one we considered, machine learning can already start delivering useful results. Figure 2: Four temperature profiles from the cross-validation dataset (solid red lines) and the corresponding predictions from different models (dashed lines).OutlookWe are currently exploring how initial conditions impact the time a system takes to reach a statistically-steady state in terms of temperature and RMS velocity. We're comparing four starting points: hot, cold, linear temperature profiles, and predictions from our neural network. This will quantify the efficiency gain achieved by using the NN predictions to accelerate simulations towards equilibrium. In addition, we also plan to share the dataset and trained neural network with the community.

An increasing number of applications of dynamic neural networks has been developed for digital signal processing (DSP), dynamic neural networks are feedforward neural networks with commonly used scalar synapses replaced by linear filters. This provides feedforward neural networks with the capability of performing dynamic mappings, which depend on past input values, dynamic neural networks are suitable for time series prediction, nonlinear system identification, and signal processing applications. Their most popular types are Finite Impulse Response (FIR) neural networks, which are obtained by replacing synapses with finite impulse response filters. Due to their guaranteed stability characteristic and easy to minimize error surface they have been used with great success in many applications such as signal enhancement, noise cancellation, classification of input patterns, system identification, prediction, and control.. Most of the works on system identification using neural networks are based on multilayer feedforward neural networks with backpropagation learning or more efficient variations of this algorithm, an elegant method for training layered networks. This paper is based on work in a Dynamic System Modeling (DYSMO) and as an application for speed control of DC motor drive.

The learning speed of the feed forward neural network takes a lot of time to be trained which is a major drawback in their applications since the past decades. The key reasons behind may be due to the slow gradient-based learning algorithms which are extensively used to train the neural networks or due to the parameters in the networks which are tuned iteratively using some learning algorithms. Thus, in order to eradicate the above pitfalls, a new learning algorithm was proposed known as Extreme Learning Machines (ELM). This algorithm tries to compute Hidden-layer-output matrix that is made of randomly assigned input layer and hidden layer weights and randomly assigned biases. Unlike the other feedforward networks, ELM has the access of the whole training dataset before going into the computation part. Here, we have devised a new two-layer-feedforward network (TFFN) for ELM in a new manner with randomly assigning the weights and biases in both the hidden layers, which then calculates the output-hidden layer weights using the Moore-Penrose generalized inverse. TFFN doesn't restricts the algorithm to fix the number of hidden neurons that the algorithm should have. Rather it searches the space which gives an optimized result in the neurons combination in both the hidden layers. This algorithm provides a good generalization capability than the parent Extreme Learning Machines at an extremely fast learning speed. Here, we have experimented the algorithm on various types of datasets and various popular algorithm to find the performances and report a comparison.

This paper proposes a system applying a pyramid neural network for classifying the hepatic parenchymal diseases in ultrasonic B-scan texture. The conventional multilayer neural network emphasizing on the data carried by the last hidden layer has the drawback of not fully utilizing the information carried by the input data. A pyramid network can solve the problem successfully. To solve the common problem of neural network, which is time-consuming in computation, FDWT (fast discrete wavelet transform) is used as a key technique for preprocessing to cut down the size of patterns feed to the network. The B-scan patterns are wavelet transformed, and then the compressed data is fed into a pyramid neural network to diagnose the type of cirrhotic diseases. The performance of the proposed system and a system based on the conventional multilayer network architecture with is compared. The result shows that compared with the conventional 3-layer neural network, the performance of the proposed pyramid neural network is improved by effectively utilizing the lower layer of the neural network. The performance is examined in a series of computer simulations. It is shown that the proposed system, a pyramid neural network with 2 hidden layers, is suitable for diagnosis of cirrhosis

Neural networks are efficient and effective in predicting system states in hydrology. However, most current approaches lack hydrological flow partitioning, do not allow for training on measurements of multiple variables, or lack capability to tightly integrate physically-based components. To address these shortcomings I propose and evaluate an approach referred to below as Dynamical System Neural Network (DSNN). DSNN is a feedforward neural network with an architecture that resembles the organisation in components of the real-world system it represents. In hydrology, the DSNN represents each water flow (e.g. seepage, snow melt) by a collection of input, hidden, and output neural layers, where each input is the state of a hydrological storage (e.g. groundwater storage influencing seepage) or other variable (e.g. air temperature influencing snow melt). These components are interconnected to form a single neural network of the complete dynamical system considered, where all storages and flows are explicitly quantified. If physical understanding of a flow and its parameterization is available, a known formulation can be used as a replacement of a neural network component. The DSNN is applied forward in time, backpropagating gradients over all timesteps. It can be run in spatially lumped or semi-distributed mode. To demonstrate the approach, a DSNN is presented of the Austrian Dorfertal (Kals) Alpine catchment containing snow and subsurface water storages and associated flows including streamflow. The DSNN is trained, validated, and tested on daily streamflow over ~40 years. To explore the capability of the DSNN in estimating the magnitude and dynamics of internal system storages (snow water equivalent, subsurface water storage) and flows (evapotranspiration, sublimation, snowmelt, seepage), the DSNN is first trained and tested with streamflow data generated by a conceptual model. The DSNN turns out to be capable of reproducing - with a satisfactory level of precision - the system states and fluxes calculated by the conceptual model, with decreasing performance when measurement error is added to the artificially generated streamflow data before training. To explore its predictive performance, the DSNN is applied on measured streamflow data for the Dorfertal, comparing multiple DSNN setups that represent all flows as neural network components or only a subset of flows where remaining flows are represented with a standard conceptual model (e.g. linear reservoir). Preliminary results indicate that in predictive performance, in most setups, the DSNN outperforms a standard conceptual model trained on the same streamflow data, with NSE values for testing of 0.74 and 0.71, respectively. This preliminary result indicates DSNN to be a promising approach for blending process-based and neural network based modelling as well as for training (i.e. calibration) of neural network models on measurements of multiple hydrological variables as these are all explicitly represented by the DSNN and can thus be incorporated in the loss function (e.g. streamflow, snow depth, groundwater, evapotranspiration).

Besides the feedforward neural networks, there are the recurrent networks, where the impulses can be transmitted in both directions due to some reaction connections in these networks. Recurrent Neural Networks (RNNs) are linear or nonlinear dynamic systems. The dynamic behavior presented by the recurrent neural networks can be described both in continuous time, by differential equations and at discrete times by the recurrence relations (difference equations). The distinction between recurrent (or dynamic) neural networks and static neural networks is due to recurrent connections both between the layers of neurons of these networks and within the same layer, too. The aim of this chapter is to describe a Recurrent Fuzzy Neural Network (RFNN) model, whose learning algorithm is based on the Improved Particle Swarm Optimization (IPSO) method.

In many applications neural networks must process or generate time series, and various network paradigms exist for this purpose. Two prominent examples are time-delay neural networks (TDNN), which are known for their noise suppression capability, and NARX (nonlinear autoregressive models with exogenous inputs) networks, which have a powerful modeling ability (at least Turing equivalence). In this article, we suggest a combination of these two approaches, called dynamic neural network (DYNN), which unifies the particular advantages. Efficient training algorithms are needed to adjust the weights of DYNN. Here, we describe an algorithm for the computation of first-order information about the error surface: temporal backpropagation through time (TBPTT). Essentially, this algorithm is a combination of temporal backpropagation (used for TDNN) and backpropagation through time (used for NARX). The first-order information is then utilized to apply the scaled conjugate gradient (SCG) learning algorithm which approximates second-order with first-order information. The benefits of this approach are shown by means of two benchmark data sets: logistic map and building. It is shown that SCG for DYNN is significantly faster and more accurate than other learning algorithms (e.g. TBPTT, resilient propagation, memoryless Quasi-Newton).

High pressure air compressor valve fault diagnosis using feedforward neural networks

The development of accurate models to describe and predict pressure inactivation kinetics of microorganisms is very beneficial to the food industry for optimization of process conditions. The need for “intelligent” methods to model highly nonlinear systems is long established. Feed-forward neural networks have been successfully used for modeling of nonlinear systems. The objective of this research is to investigate the capabilities of a new wavelet neural network, to predicting of survival curves of Listeria monocytogenes inactivated by high hydrostatic pressure in UHT whole milk. The performance of the proposed scheme has been compared against a dynamic neural network and classic statistical models used in food microbiology.

The history of artificial neural networks goes back to the 1950s. Many various concepts have been proposed since then. At the beginning, there was a simple linear element, Adaline, Madaline, and Perceptron. Then the feedforward network with the backpropagation opened a new era of artificial intelligence. Now, the catalogue of neural networks is quite rich including many recurrent networks (NN) like Hopfield NN, Boltzmann machine, restricted Boltzmann machine, dynamic neural networks and deep neural networks. The most spectacular results are obtained using a few types of deep convolutional networks.

Artificial intelligent provides diverse solutions for the complex problems in agriculture research. The study aimed to use three models of artificial neural networks (Feed Forward Neural Network (FFNN), Generalized Regression Neural Network (GRNN) and Radial-Basis Neural Network (RBNN)) in the field of wheat yield prediction. 27-year data for the period (1986-2012) were utilized to improve the models and four-year data (2013 and 2016) were used to estimate the models, to compare their outputs with the measured data. Prediction data was not entered in the process of building neural network models. The results showed that the optimal configuration of the FFNN model consists of 40 neurons in the hidden layer (8-40-1). The Tan Sigmoid activation function was used in both the hidden layer and the output layer using all of these models (anterior neural feeding network and the regression neural network and radial base neural network) in the 4-year wheat yield forecast field for production (2013-2016) by applying 8 input parameters that were result of NMMS (8.6%, 7.6% and 15.7% resp.), To find that FFNN and GRNN provide the best result from BRNN because while the information set was large or in a wide range, then the range data ranges from -1 to +1 (normalization data) , GRNN gives better outcomes after the information or sample data were in large range.

This paper deals with the adaptive nonlinear identification and trajectory tracking via dynamic multilayer neural network (NN) with different timescales. Two NN identifiers are proposed for nonlinear systems identification via dynamic NNs with different timescales including both fast and slow phenomenon. The first NN identifier uses the output signals from the actual system for the system identification. In the second NN identifier, all the output signals from nonlinear system are replaced with the state variables of the NNs. The online identification algorithms for both NN identifier parameters are proposed using Lyapunov function and singularly perturbed techniques. With the identified NN models, two indirect adaptive NN controllers for the nonlinear systems containing slow and fast dynamic processes are developed. For both developed adaptive NN controllers, the trajectory errors are analyzed and the stability of the systems is proved. Simulation results show that the controller based on the second identifier has better performance than that of the first identifier.

Arabic language can be used by native and non- native speakers; due to Arabic is the language of the holy book of Muslims. In this paper, Arabic phoneme recognition system is proposed based on Malay speakers. This system consists of three main stages. The first stage is noise reduction and it aims to enhance the phoneme signals by excluding the unvoiced signals and keep only the voiced signal. Wiener filter is adapted to accomplish this task. The second stage is based on Mel-Frequency Cepstral Coefficients method to extract a vector of features to represent each phoneme signal. Eventually, pattern recognition neural network is designed as recognizer. The proposed system produces sufficient outcomes with 20 hidden neurons. Keywords—Arabic; Malay; pattern recognition; Wiener; Mel- Frequency Cepstral Coefficients I. INTRODUCTION It is well known the wide range of Artificial Neural Networks (ANN) application in speech recognition, financial, telecommunications, electronics and medical. As for speech recognition system for Arabic language, (1) & (2) have addressed the implementation of recognition system based on Arabic native speakers. In general, neural networks (NNs) for phonemes recognition are divided into several categories for instance Very Large Scale Integration (VLSI) NN that can be divided into digital and analogue with digital NNs being more compatible with feed-forward neural networks while analogue NNs are found to be more successful with recurrent NNs (3). From Automatic Speech System (ASR) point of view, NNs for recognition purpose can be divided into two main category namely conventional neural networks such as Multilayer Perceptron (MLP) and Radial Basis Function (RBF) and secondly Recurrent Neural Networks (RNN). The first category of NNs was implemented as pattern classifiers and proven capable for recognition purpose but not at par as compared to Hidden Markov Model (HMM) performance (2). At present, HMM has been proven to be the most successful approach in ASR research area until recently the possibility to hybrid both HMM and ANN (4) & (5). In general, speech recognition systems consist of three stages specifically noise reduction, feature extraction and classification. The aim of noise reduction is to reduce the level of noise in the speech signal and make it more compatible for the ASR. As for feature extraction, this process attempted to extract the most relevant features related to the phoneme signal. The extracted features acted as input features of the NN during training and testing. Finally, the purpose of classification process is to arrange the most similar and related features in categories based on the training phase. Basically, networks training database are being observed and updated during training session of the networks, involving the weights and biases arguments. Networks with lower errors can be considered as better networks (6). Hence, this paper deems to explore further the ability of NN for Arabic phonemes recognition spoken by Malay individuals. This paper is arranged as following; related work is discussed in section II followed by section III that discussed method to be implemented based on Wiener filter, Mel-Frequency Cepstral Coefficients and pattern recognition neural network. Section IV elaborated experimental analysis and results and followed by Conclusion section.

Neural network, a nonalgorithmic, nondigital, intensely parallel and distributive information processing system, is being used more and more everyday. The main interest in neural networks is rooted in the recognition that the human brain processes information in a different manner than conventional digital computers. Computers are extremely fast and precise at executing sequences of instructions that have been formulated for them. A human information processing system is composed of neurons switching at speeds about a million times slower than computer gates. Yet, humans are more efficient than computers at such computationally complex tasks as speech and other pattern-recognition problems. Artificial neural systems, or neural networks, are physical cellular systems that can acquire, store, and use experiential knowledge. The knowledge is in the form of stable states or mapping embedded in networks that can be recalled in response to the presentation of cues. In a typical neural data processing procedure, the database is divided into two separate portions called training and test sets. The training set is used to develop the desired network. In this process (depending on the paradigm that is being used), the desired output in the training set is used to help the network learn by adjusting the weights between its neurons or processing elements. Once the network has learned the information in the training set and has"converged," the test set is applied to the network for verification. It is important to note that although the user has the desired output of the test set, it has not been seen by the network. This ensures the integrity and robustness of the trained network. A handful of articles on the use of neural networks in the petroleum industry has appeared in SPE conferences, proceedings, and publications in the past 2 years. These articles can be divided into two categories: those that use neural networks to analyze formation lithology from well logs and those that use neural networks to pick a reservoir model to be used in conventional welltest interpretation studies. These tasks are usually done by log analysts and reservoir engineers, and their automation using a fault-tolerant process may prove valuable. Neural networks can help engineers and researchers by addressing some fundamental petroleum engineering problems as well as specific ones that conventional computing has been unable to solve. Petroleum engineers may benefit from neural networks on occasions when engineering data for design and interpretations are less than adequate. This is an especially common occurrence in the Appalachian basin, where some fields are quite old. Lack of adequate engineering data may also be encountered because of the high cost of coring, well testing, and so on. Neural networks have shown great potential for generating accurate analysis and results from large amounts of historical data that otherwise would seem not to be useful or relevant in the analysis. An example of such a problem was encountered by a gas company for a gas storage field in Ohio. In the absence of appropriate data, which normally would make engineering design and evaluation of the fracturing jobs virtually impossible, a carefully designed neural network was able to predict the performance of fracturing jobs with great accuracy. A linear plot of the actual fracturing job results (data never seen by the network during training) and network predictions resulted in a correlation coefficient of 0.98, where 1.00 is a perfect match. Neural networks have proved to be valuable pattern-recognition tools. They are capable of finding highly complex patterns within large amounts of data. A relevant example is well log interpretation. It is generally accepted that there is more information embedded in well logs than meets the eye. Determination, prediction, or estimation of formation permeability without actual laboratory measurement of the cores or interruption in production for well test data collection has been a fundamental problem for petroleum engineers. From geophysical well log data, it was possible to predict and/or estimate permeability of a highly heterogeneous formation in West Virginia, as shown in Fig 1. Reference Mohaghegh, S., Arefi, R., and Ameri, S.: "Design and Development of an Artificial Neural Network for Prediction of Formation Permeability,"paper SPE 28237 presented at the 1994 SPE Petroleum Computer Conference, July31-Aug. 3, Dallas.