Multiresolution locally expanded HONN for handwritten numeral recognition

In 1984 Hopfield showed that the time evolution of a symmetric Hopfield neural networks are a motion in state space that seeks out minima in the energy function (i.e., equilibrium point set of Hopfield neural networks). Because high-order Hopfield neural networks have more extensive applications than Hopfield neural networks, and have been discussed on the convergence of the networks. In practice, a neural network is often subject to environmental noise. It is therefore useful and interesting to find out whether the high-order neural network system still approacher some limit set under stochastic perturbation. In this paper, we will give a number of useful bounds for the noise intensity under which the stochastic high-order neural network will approach its limit set. Our result cancels the requirement of symmetry of the connection weight matrix and includes the classic result on Hopfield neural networks, which is a special case of stochastic high-order Hopfield neural networks. In the end, A example is given to verify the effective of our results.

This chapter develops a new nonlinear model, Ultra high frequency Trigonometric Higher Order Neural Networks (UTHONN), for time series data analysis. Results show that UTHONN models are 3 to 12% better than Equilibrium Real Exchange Rates (ERER) model, and 4 – 9% better than other Polynomial Higher Order Neural Network (PHONN) and Trigonometric Higher Order Neural Network (THONN) models. This study also uses UTHONN models to simulate foreign exchange rates and consumer price index with error approaching 0.0000%.

The data in the real world are complex and varied. When we apply artificial neural networks to simulate the real-world commerce data through simple functions, accuracy could become problematic. To overcome this issue, we could choose Higher Order Neural Network (HONN) models, which can simulate a data set very accurately. This paper has developed a new open-box HONN model with fractional functions, called a Fractional High Frequency Cosine and Sine HONN (FHFCSHONN) model. FHFCSHONN structure and learning algorithm formulas are studied too. This paper also built a new software package called a FHFCSHONN simulator. Based on the commerce data testing results, the FHFCSHONN model's average error is 0.6980%, while the other three HONN model average errors are 4.0160%, 4.2370%, and 4.3556% respectively.

An Artificial Neural Network (ANN) works by creating connections between different processing elements (artificial neurons). ANNs have been extensively used for Data Mining, which extracts hidden patterns and valuable information from large databases. This paper introduces a new adaptive Higher Order Neural Network (HONN) model and applies it in data mining tasks such as determining breast cancer recurrences and predicting incomes base on census data. An adaptive hyperbolic tangent function is used as the neuron activation function for the new adaptive HONN model. The paper compares the new HONN model against a Multi-Layer Perceptron (MLP) with the sigmoid activation function, an RBF Neural Network with the Gaussian activation function, and a Recurrent Neural Network (RNN) with the sigmoid activation function. Experimental results show that the new adaptive HONN model offers several advantages over conventional ANN models such as better generalisation capabilities as well as abilities in handling missing values in a dataset.

Time series forecasting gets much attention due to its impact on many practical applications. Higher-order neural network with recurrent feedback is a powerful technique which used successfully for forecasting. It maintains fast learning and the ability to learn the dynamics of the series over time. In general, the most used recurrent feedback is the network output. However, no much attention has been paid to use network error instead of the network output. For that, in this paper, we propose a novel model which is called Ridge Polynomial Neural Network with Error Feedback (RPNN-EF) that combines the properties of higher order and error feedback recurrent neural network. Three signals have been used in this paper, namely heat wave temperature, IBM common stock closing price and Mackey–Glass equation. Simulation results show that RPNN-EF is significantly faster than other RPNN-based models for one-step ahead forecasting and its forecasting performance is more significant than these models for multi-step ahead forecasting.

In present days, real time navigation depends on Kalman filter to fuse data from Global Positioning System (GPS) and Inertial Navigation System (INS). But there exist drawbacks like long design time, requirement of prior knowledge in fusing data from GPS and INS using Kalman Filter. In order to overcome the drawbacks of Kalman filter, GPS and INS integration are done using Artificial Neural Networks (ANN) like FFNNs. But still there exists certain drawbacks like less accuracy in FFNNs for GPS and INS data integration. Therefore, this paper introduces Higher Order Neural Networks (HONNs) approach for INS and GPS data integration to overcome the drawback. A higher order feed forward neural network architecture with optimum number of nodes is used. In the given architectures, the replacement of summation at each node by multiplication results in more powerful mapping, because of its capability of processing higher-order information from training data. Performance comparison of HONNs with FFNNs shows that the proposed architecture provides satisfactory results in terms of error rate and number of epochs. The HONNs like Multiplicative Neural Network (MNN) module and Sigma-Pi-ANN module and Traditional Feed Forward Neural Networks(FFNNs) like Radial Basis Function Neural Network (RBF) module and Back propagation Neural Network (BPN) module are trained to predict the INS position error and provide accurate position of the moving aircrafts. Sigma-Pi ANN is found to be the best in terms of accuracy, number of epochs and execution time among the two HONNs.

This chapter develops a new nonlinear model, Ultra high frequency siGmoid and Trigonometric Higher Order Neural Networks (UGT-HONN), for data pattern recognition. UGT-HONN includes Ultra high frequency siGmoid and Sine function Higher Order Neural Networks (UGS-HONN) and Ultra high frequency siGmoid and Cosine functions Higher Order Neural Networks (UGC-HONN). UGS-HONN and UGC-HONN models are used to recognition data patterns. Results show that UGS-HONN and UGC-HONN models are better than other Polynomial Higher Order Neural Network (PHONN) and Trigonometric Higher Order Neural Network (THONN) models, since UGS-HONN and UGC-HONN models to recognize data pattern with error approaching 0.0000%.

This chapter proposes a theoretical framework for parallel implementation of Deep Higher Order Neural Networks (HONNs). First, we develop a new partitioning approach for mapping HONNs to individual computers within a master-slave distributed system (a local area network). This will allow us to use a network of computers (rather than a single computer) to train a HONN to drastically increase its learning speed: all of the computers will be running the HONN simultaneously (parallel implementation). Next, we develop a new learning algorithm so that it can be used for HONN learning in a distributed system environment. Finally, we propose to improve the generalisation ability of the new learning algorithm as used in a distributed system environment. Theoretical analysis of the proposal is thoroughly conducted to verify the soundness of the new approach. Experiments will be performed to test the new algorithm in the future.

Business is a diversified field with general areas of specialisation such as accounting, taxation, stock market, and other financial analysis. Artificial Neural Networks (ANN) have been widely used in applications such as bankruptcy prediction, predicting costs, forecasting revenue, forecasting share prices and exchange rates, processing documents and many more. This chapter introduces an Adaptive Higher Order Neural Network (HONN) model and applies the adaptive model in business applications such as simulating and forecasting share prices. This adaptive HONN model offers significant advantages over traditional Standard ANN models such as much reduced network size, faster training, as well as much improved simulation and forecasting errors, due to their ability to better approximate complex, non-smooth, often discontinuous training data sets. The generalisation ability of this HONN model is explored and discussed.

Target Recognition Based on Directional Filter Banks and Higher-Order Neural Networks

In this paper, effective multiresolution image representations using a combination of 2-D filter bank (FB) and directional wavelet transform (WT) are presented. The proposed methods yield simple implementation and low computation costs compared to previous 1-D and 2-D FB combinations or adaptive directional WT methods. Furthermore, they are nonredundant transforms and realize quad-tree like multiresolution representations. In applications on nonlinear approximation, image coding, and denoising, the proposed filter banks show visual quality improvements and have higher PSNR than the conventional separable WT or the contourlet.

Gaussian and Laplacian pyramids have long been important for image analysis and compression. More recently, multiresolution pyramids have become an important component of machine learning and deep learning for image analysis and image recognition. Constructing Gaussian and Laplacian pyramids consists of a series of filtering, decimation, and differencing operations, and the quality indicator is usually mean squared reconstruction error in comparison to the original image. We present a new characterization of the information loss in a Gaussian pyramid in terms of the change in mutual information. More specifically, we show that one half the log ratio of entropy powers between two stages in a Gaussian pyramid is equal to the difference in mutual information between these two stages. We show that this relationship holds for a wide variety of probability distributions and present several examples of analyzing Gaussian and Laplacian pyramids for different images.

Motor drives are traditionally designed with relatively inexpensive analog components. The weaknesses of analog systems are their susceptibility to temperature variations and the component aging. Another difficulty is upgrading the system for new needs. Digital control structures eliminate drifts, solve complex mathematical equations and, by using a programmable processor, can be easily upgraded for new control algorithms. Digital signal processors (DSP) go further; their high performance allows them to perform high resolution control and minimize control loop delays. By the help of their fast processor which works parallel, many complicated control algorithms can be implemented easily. With the importance of digital control structures, it was used a TMS320C50 processor board to implement the dynamic speed controller to induction motor. Also, it was used to increase the performance a HONN (high order neural network) structure in the ANN (artificial neural network) architecture in a very short time.

The detection of gaussian signals with unknown correlation coefficient, ρ s is considered. A strategy for designing high order neural networks (HONN) in composite hypothesis test is proposed. A HONN trained with ρ s varying uniformly in [0, 1] is considered to approximate the Average Likelihood Ratio (ALR). In order to compare the suitability of the approximation, a sub-optimal solution based on constrained generalized likelihood ratio is used. A study of the computational cost is carried out. Results show that a HONN is able to approximate the ALR with a low computational cost.

The Sixth International Symposium on Neural Networks (ISNN 2009)