Neural Networks, Bayesian a posteriori Probabilities, and Pattern Classification

Instantaneous learning is a desirable feature in neural networks. This type of learning enables the network to be trained very quickly, typically in just one or two passes of the training set as opposed to hundreds or even thousands of passes for networks trained by an iterative process, such as the error backpropagation (BP) algorithm. This dissertation first reviews several existing types of neural networks with instantaneous learning capability. It then proposes a new network called the Fuzzy Classification (FC) Neural Network that can be trained with two passes of the training samples. The first pass assigns the synaptic weights for the input and output layers. The second pass determines the radius of generalization r for each training sample. The network exhibits fuzziness in two regards: (1) by fuzzification of the location of each training vector in the input space; and (2) by assigning fuzzy memberships of output classes to new input vectors. The operation of the FC network is analyzed from different perspectives, namely, separability of patterns, curve fitting and kernel regression. It is shown that one mode of operation of the FC network can be made to behave like a Radial Basis Function (RBF) network by proper choice of membership function and network parameter. The performance of the FC network as a pattern classifier is also discussed in a statistical framework. The generalization performance of the FC network is compared against that of the CC4 Comer Classification Neural Network, the RBF network, and the Multilayer Perceptron network trained by the BP algorithm. Experiments involving a Henon map time series, a Mackey-Glass time series, and a spiral pattern are used in the comparison. The performance of the FC network is found to be comparable to that of RBF and BP networks, and better than that of the CC4 network. Its performance is also more scalable than those of CC4 and RBF networks. Further, the time taken to design an FC network for any given problem is much shorter compared to the other three networks.

Bayesian model updating provides a rigorous framework to account for uncertainty induced by lack of knowledge about engineering systems in their respective mathematical models through updates of the joint probability density function (PDF), the so-called posterior PDF, of the unknown model parameters. The Markov chain Monte Carlo (MCMC) methods are currently the most popular approaches for generating samples from the posterior PDF. However, these methods often found wanting when sampling from difficult distributions (e.g., high-dimensional PDFs, PDFs with flat manifolds, multimodal PDFs, and very peaked PDFs). This paper introduces a new multi-level sampling approach for Bayesian model updating, called Sequential Gauss-Newton algorithm, which is inspired by the Transitional Markov chain Monte Carlo (TMCMC) algorithm. The Sequential Gauss-Newton algorithm improves two aspects of TMCMC to make an efficient and effective MCMC algorithm for drawing samples from difficult posterior PDFs. First, the statistical efficiency of the algorithm is enhanced by use of the systematic resampling scheme. Second, a new MCMC algorithm, called Gauss-Newton MCMC algorithm, is proposed which is essentially an M-H algorithm with a Gaussian proposal PDF tailored to the posterior PDF using the gradient and Hessian information of the negative log posterior. The effectiveness of the proposed algorithm for solving the Bayesian model updating problem is illustrated using three examples with irregularly shaped posterior PDFs.

This paper presents a method for analog circuit fault diagnosis by using neural networks. This method exploits a DC approach for constructing a dictionary in fault diagnosis using the neural network's classification capability. Also, Radial Basis Function (RBF) and backward error propagation (BEP) networks are considered and compared for analog fault diagnosis. The primary focus of the paper is to provide robust diagnosis using a mechanism to deal with the problem of component tolerance and reduce testing time. Simulation results show that the radial basis function network with reasonable dimension has double precision in fault classification but its classification is local, while the backward error propagation network with reasonable dimension has single precision in fault classification but its classification is global.

Pattern classification by a condensed neural network

The author extends a previous review and focuses on feed-forward neural-net classifiers for static patterns with continuous-valued inputs. He provides a taxonomy of neural-net classifiers, examining probabilistic, hyperplane, kernel, and exemplar classifiers. He then discusses back-propagation and decision-tree classifiers; matching classifier complexity to training data; GMDH (generalized method of data handling) networks and high-order nets; K nearest-neighbor classifiers; the feature-map classifier; the learning vector quantizer; hypersphere classifiers; and radial-basis function classifiers.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Three learning phases for radial-basis-function networks

In this study, the relationship between nonlinear model properties and inverse problem solutions is analyzed using a numerical technique based on the inverse problem theory formulated by Mosegaard and Tarantola. According to this theory, the inverse problem and solution are defined via convolution and conjunction of probability density functions (PDFs) that represent stochastic information obtained from the model, observations, and prior knowledge in a joint multidimensional space. This theory provides an explicit analysis of the nonlinear model function, together with information about uncertainties in the model, observations, and prior knowledge through construction of the joint probability density, from which marginal solution functions can then be evaluated. The numerical analysis technique derived from the theory computes the component PDFs in discretized form via a combination of function mapping on a discrete grid in the model and observation phase space and Monte Carlo sampling from known parametric distributions. The efficacy of the numerical analysis technique is demonstrated through its application to two well-known simplified models of atmospheric physics: damped oscillations and Lorenz’s three-component model of dry cellular convection. The major findings of this study include the following: (i) Use of a nonmonotonic forward model in the inverse problem gives rise to the potential for a multimodal posterior PDF, the realization of which depends on the information content of the observations and on observation and model uncertainties. (ii) The cumulative effect of observations over time, space, or both could render the final posterior PDF unimodal, even with the nonmonotonic forward model. (iii) A greater number of independent observations are needed to constrain the solution in the case of a nonmonotonic nonlinear model than for a monotonic nonlinear or linear forward model for a given number of degrees of freedom in control parameter space. (iv) A nonlinear monotonic forward model gives rise to a skewed unimodal posterior PDF, implying a well-posed maximum likelihood inverse problem. (v) The presence of model error greatly increases the possibility of capturing multiple modes in the posterior PDF with the nonmonotonic nonlinear model. (vi) In the case of a nonlinear forward model, use of a Gaussian approximation for the prior update has a similar effect to an increase in model error, which indicates there is the potential to produce a biased mean central estimate even when observations and model are unbiased.

In order to forecast missile storage reliability better, the paper researched a forecasting method based on neural network which is with the ability of actualizing multi-nonlinear mapping from input to output, and discussed steps of forecasting based on back propagation (BP) network and radial basis function (RBF) network respectively. At last, the storage reliability of one type ship-to-ship missile is forecasted based on BP network and RBF network respectively. The results show that both of the BP and RBF are suitable for Missile storage reliability forecasting, and the precision of the train goal is better by using RBF network. RBF network is more suitable for dealing with this problem.

Although flexible neural networks (FNNs) have been used more successfully than classical neural networks (CNNs), nothing is rigorously known about their properties. In fact, they are not even well known to the systems and control community. In this paper, theoretical evidence is given for their superiority over CNNs. Following an overview of flexible bipolar sigmoid functions (FBSFs), several fundamental properties of feedforward and recurrent FNNs are established. For the feedforward case, it is proven that similar to CNNs, FNNs with as few as a single hidden layer (SHL) are universal approximators. It is also proven that unlike irreducible SHL CBSNNs, irreducible SHL FBSNNs are nonuniquely determined by their input-output (I-0) maps, up to a finite group of symmetries. Then, recurrent FNNs are introduced. It is observed that they can be interpreted as a generalization of the conventional state-space framework. For the recurrent case, it is substantiated that similar to CBSNNs, FBSNNs are universal approximators. Necessary and sufficient conditions for the controllability and observability of a generic class of them are established. For a subclass of this class, it is proven that unlike CBSNNs, FBSNNs are nonuniquely determined by their I-O maps, up to a finite group of symmetries, and that every system inside this subclass is minimal. Finally, a new class of FNNs, namely, flexible bipolar radial basis neural networks (FBRBNNs) is introduced. It is proven that as in the case of classical radial basis neural networks (CRBNNs), feedforward SHL FBRBNNs are universal approximators.

Best estimate plus uncertainty for the safety assessment of nuclear power plant transient requires, among others, estimating the probability density function (PDF) of physical model parameters in thermal-hydraulic system codes. In that context, Bayesian calibration based on experimental data from separate-effect test facilities are increasingly popular to inform the PDF of a single thermal-hydraulic phenomenon. These calibrations are, however, time intensive, especially when considering multiple time-dependent outputs. Calibrating on many tests with different boundary conditions and potentially different phenomena to derive PDFs applicable to complex transients appears intractable, even using hierarchical modeling. In this paper, we start investigating this problem by considering a set of Flooding Experiments with Blocked Arrays reflood tests with different boundary conditions. We use TRACE v5.0p3 to model time- and space-dependent temperature profiles, pressure drops, and liquid carry-over. Global sensitivity analysis helps screen out noninfluential parameters and gain a detailed understanding of the modeled physics of reflood. The analysis shows that, for all tests, the outputs were sensitive to a similar set of influential model parameters. In turn, Bayesian calibration yields similar posterior PDFs for the influential parameters, and forward propagation of these posterior PDFs yields similar confidence intervals. As such, the information of the investigated tests can well be represented by a unique posterior PDF. Such simplifications, although not general, are welcome to help manage the intensive calibration effort necessary for dealing with complex thermal-hydraulic transients encountered in nuclear power plants.

In this chapter, the CIP is formulated in a statistical framework. Byusing the Bayesian inference theory, we cast the CIP asaninformation transfer problem, in which the prior information andtheinformation transferred from state observations are combinedtoreduce uncertainty in the estimated parameters. The priorinformationis modeled using a probability density function (PDF)called the priorPDF and the inverse solution is also a PDF known asthe posteriorPDF. Because it is a PDF, the inverse solution is alwaysexistent andunique but with uncertainty. When the posterior PDF is ina relativelysimple form, point estimates of the unknown parameterscan bereadily obtained by solving an optimization problem, just as wehavedone in the deterministic framework. When the posterior PDFhas acomplex multimodal shape, however, the non-uniquenessandinstability issues associated with the inverse solution arise again.Forsuch cases, Monte Carlo sampling methods provide powerfultoolsfor learning the posterior PDFs without requiring theiractualfunctional forms be known. Two popular Markov Chain MonteCarlo(MCMC) algorithms are introduced. The application of MCMCforinverse solution and global optimization is also discussed.

Arabic language can be used by native and non- native speakers; due to Arabic is the language of the holy book of Muslims. In this paper, Arabic phoneme recognition system is proposed based on Malay speakers. This system consists of three main stages. The first stage is noise reduction and it aims to enhance the phoneme signals by excluding the unvoiced signals and keep only the voiced signal. Wiener filter is adapted to accomplish this task. The second stage is based on Mel-Frequency Cepstral Coefficients method to extract a vector of features to represent each phoneme signal. Eventually, pattern recognition neural network is designed as recognizer. The proposed system produces sufficient outcomes with 20 hidden neurons. Keywords—Arabic; Malay; pattern recognition; Wiener; Mel- Frequency Cepstral Coefficients I. INTRODUCTION It is well known the wide range of Artificial Neural Networks (ANN) application in speech recognition, financial, telecommunications, electronics and medical. As for speech recognition system for Arabic language, (1) & (2) have addressed the implementation of recognition system based on Arabic native speakers. In general, neural networks (NNs) for phonemes recognition are divided into several categories for instance Very Large Scale Integration (VLSI) NN that can be divided into digital and analogue with digital NNs being more compatible with feed-forward neural networks while analogue NNs are found to be more successful with recurrent NNs (3). From Automatic Speech System (ASR) point of view, NNs for recognition purpose can be divided into two main category namely conventional neural networks such as Multilayer Perceptron (MLP) and Radial Basis Function (RBF) and secondly Recurrent Neural Networks (RNN). The first category of NNs was implemented as pattern classifiers and proven capable for recognition purpose but not at par as compared to Hidden Markov Model (HMM) performance (2). At present, HMM has been proven to be the most successful approach in ASR research area until recently the possibility to hybrid both HMM and ANN (4) & (5). In general, speech recognition systems consist of three stages specifically noise reduction, feature extraction and classification. The aim of noise reduction is to reduce the level of noise in the speech signal and make it more compatible for the ASR. As for feature extraction, this process attempted to extract the most relevant features related to the phoneme signal. The extracted features acted as input features of the NN during training and testing. Finally, the purpose of classification process is to arrange the most similar and related features in categories based on the training phase. Basically, networks training database are being observed and updated during training session of the networks, involving the weights and biases arguments. Networks with lower errors can be considered as better networks (6). Hence, this paper deems to explore further the ability of NN for Arabic phonemes recognition spoken by Malay individuals. This paper is arranged as following; related work is discussed in section II followed by section III that discussed method to be implemented based on Wiener filter, Mel-Frequency Cepstral Coefficients and pattern recognition neural network. Section IV elaborated experimental analysis and results and followed by Conclusion section.

The design of pattern classifiers such as multiprototype classifiers and neural network classifiers such as learning vector quantization and radial basis function neural networks requires reducing the size of the training data sets. In addition, memory storage, computation complexity and time, and data redundancy demand many pattern classifiers to use a smaller subset of a training data set. In this paper, we present a data reduction algorithm which automatically selects the subset of training data that faithfully represents the training data set for pattern classification. The applicability of this algorithm is demonstrated through k-nearest neighbor and learning vector quantization neural networks classifiers using both speech and synthetic data sets.

Uncertainty assessment of flow and contaminant transport in the vadose zone entails probability density functions (PDFs) of soil hydraulic parameters. An unconventional maximum likelihood (ML) approach was used in this study to estimate the PDFs of water retention parameters (e.g., van Genuchten α and m) for a situation common in field‐scale modeling where core samples are sparse and prior PDFs of the parameters are unknown. In this situation, the unconventional ML approach approximates the PDFs as multivariate Gaussian. This study developed a method of estimating the mean and covariance of the multivariate Gaussian PDF based on the results of least square methods that can be easily obtained in practice. The developed method was applied to and evaluated through numerical simulation of unsaturated flow and tracer transport at the proposed Yucca Mountain geologic repository. Another focus of this study was to investigate the effect of uncertainty in the water retention parameters on predictive uncertainty. By comparing the predictive uncertainty before and after incorporating random water retention parameters, it was found that the random water retention parameters had limited effects on the mean predictions of the state variables including percolation flux and tracer travel time from the potential repository to the water table. Incorporating the uncertainty in the water retention parameters, however, significantly increased the magnitude and spatial extent of predictive uncertainty of the state variables. In particular, incorporating the random water retention parameters significantly changed the 5th and 95th percentiles of the tracer travel time by tens of thousands of years.

A new approach is presented to enhance fuzzy modeling using multidimensional fuzzy sets directly in the fuzzy model in place of decomposed fuzzy sets by projection. The antecedent fuzzy sets in the form of multivariable functions are represented in continuous form. This is accomplished by using a multi input and multi output neural network, which is in particular radial basis functions (RBF) network providing the required multivariable function approximation properties in a fuzzy model. The inputs of the network are fuzzy model inputs, outputs are the membership function values as to the multivariable fuzzy sets involved in the model. Thus, each output is restricted to one multivariable fuzzy set. The RBF network is trained according to the multidimensional fuzzy sets which are identified after fuzzy clustering of the data. Based on this, the linear model parameters are determined by the method of least squares in a straightforward manner. The transparency issues can be handled by means of projection process to have information about the shape and locations of the membership functions pertinent to each variable at the input. The accurate fuzzy modeling having been thus guaranteed the information on the linear model parameters are used to establish the multivariable fuzzy rules with their respective validity regions. The redundancy of the number of fuzzy sets can be circumvented by classical cluster merging algorithms available. In this approach, the fuzzy rules are determined from the local linear models, however in contrast with the conventional fuzzy modeling, the model outputs are determined by the multivariable fuzzy sets. To obtain regular sets, RBF network is trained by means of orthogonal least squares (OLS) method so that at the same time the convergence issues of general neural network training is eliminated.