Ultra High Frequency Sigmoid and Trigonometric Higher Order Neural Networks for Data Pattern Recognition

This chapter develops a new nonlinear model, Ultra high frequency Trigonometric Higher Order Neural Networks (UTHONN), for time series data analysis. Results show that UTHONN models are 3 to 12% better than Equilibrium Real Exchange Rates (ERER) model, and 4 – 9% better than other Polynomial Higher Order Neural Network (PHONN) and Trigonometric Higher Order Neural Network (THONN) models. This study also uses UTHONN models to simulate foreign exchange rates and consumer price index with error approaching 0.0000%.

New open box and nonlinear system model of Ultra High Frequency Sigmoid and Sine Artificial Higher Order Neural Network (UGS-HONN) is presented in this paper. A new learning algorithm for UGS-HONN is also developed from this study. A time series data modeling system, UGS-HONN Simulator, is built based on the UGS-HONN system models too. Test results show that average error of UGS-HONN system models are closing to zero (10-6). The average errors of Polynomial Higher Order Neural Network (PHONN), Trigonometric Higher Order Neural Network (THONN), and Sigmoid polynomial Higher Order Neural Network (SPHONN) models are from 2.8128% to 4.9076%. It means that UGS-HONN system models are 2.8128% to 4.9076% better than PHONN, THONN and SPHONN models.

The data in the real world are complex and varied. When we apply artificial neural networks to simulate the real-world commerce data through simple functions, accuracy could become problematic. To overcome this issue, we could choose Higher Order Neural Network (HONN) models, which can simulate a data set very accurately. This paper has developed a new open-box HONN model with fractional functions, called a Fractional High Frequency Cosine and Sine HONN (FHFCSHONN) model. FHFCSHONN structure and learning algorithm formulas are studied too. This paper also built a new software package called a FHFCSHONN simulator. Based on the commerce data testing results, the FHFCSHONN model's average error is 0.6980%, while the other three HONN model average errors are 4.0160%, 4.2370%, and 4.3556% respectively.

Business is a diversified field with general areas of specialisation such as accounting, taxation, stock market, and other financial analysis. Artificial Neural Networks (ANN) have been widely used in applications such as bankruptcy prediction, predicting costs, forecasting revenue, forecasting share prices and exchange rates, processing documents and many more. This chapter introduces an Adaptive Higher Order Neural Network (HONN) model and applies the adaptive model in business applications such as simulating and forecasting share prices. This adaptive HONN model offers significant advantages over traditional Standard ANN models such as much reduced network size, faster training, as well as much improved simulation and forecasting errors, due to their ability to better approximate complex, non-smooth, often discontinuous training data sets. The generalisation ability of this HONN model is explored and discussed.

An Artificial Neural Network (ANN) works by creating connections between different processing elements (artificial neurons). ANNs have been extensively used for Data Mining, which extracts hidden patterns and valuable information from large databases. This paper introduces a new adaptive Higher Order Neural Network (HONN) model and applies it in data mining tasks such as determining breast cancer recurrences and predicting incomes base on census data. An adaptive hyperbolic tangent function is used as the neuron activation function for the new adaptive HONN model. The paper compares the new HONN model against a Multi-Layer Perceptron (MLP) with the sigmoid activation function, an RBF Neural Network with the Gaussian activation function, and a Recurrent Neural Network (RNN) with the sigmoid activation function. Experimental results show that the new adaptive HONN model offers several advantages over conventional ANN models such as better generalisation capabilities as well as abilities in handling missing values in a dataset.

Ridge polynomial networks (RPNs) are a special class of high order neural networks, which are computationally much more efficient than traditional high order neural networks. Motivated by the ability of high order NNs to perform shift and rotation invariant pattern recognition tasks, we apply RPNs for function approximation and character recognition. We are using features from the image block representation of the characters and traditional invariant moments. The results are quite favoring and show that RPNs can be used effectively in pattern recognition problems.

For the task of position-, scale-, and rotation-invariant pattern recognition higher order neural networks have shown good separation results for different object classes. However, their use is limited by the large number of monomials that have to be calculated to get the invariant features. Even with the application of methods like coarse coding to reduce the number of monomials the computational requirements are still large. Here a new method is proposed where the averaging of the monomials to form the invariant features is performed adaptively with a network architecture that uses coupled nodes. Thereby the features are adapted to a specific application. This reduces significantly the number and order of the monomials necessary for the invariant recognition of patterns. Moreover, the classification part of the network becomes superfluous since the output of the network is already one-dimensional and binary. The properties of the new method in comparison to higher order neural networks are shown for position and 90-degree-rotation invariance and for invariance with respect to different object backgrounds. The application to rotations of smaller angles is outlined.

This chapter proposes a theoretical framework for parallel implementation of Deep Higher Order Neural Networks (HONNs). First, we develop a new partitioning approach for mapping HONNs to individual computers within a master-slave distributed system (a local area network). This will allow us to use a network of computers (rather than a single computer) to train a HONN to drastically increase its learning speed: all of the computers will be running the HONN simultaneously (parallel implementation). Next, we develop a new learning algorithm so that it can be used for HONN learning in a distributed system environment. Finally, we propose to improve the generalisation ability of the new learning algorithm as used in a distributed system environment. Theoretical analysis of the proposal is thoroughly conducted to verify the soundness of the new approach. Experiments will be performed to test the new algorithm in the future.

Time series forecasting gets much attention due to its impact on many practical applications. Higher-order neural network with recurrent feedback is a powerful technique which used successfully for forecasting. It maintains fast learning and the ability to learn the dynamics of the series over time. In general, the most used recurrent feedback is the network output. However, no much attention has been paid to use network error instead of the network output. For that, in this paper, we propose a novel model which is called Ridge Polynomial Neural Network with Error Feedback (RPNN-EF) that combines the properties of higher order and error feedback recurrent neural network. Three signals have been used in this paper, namely heat wave temperature, IBM common stock closing price and Mackey–Glass equation. Simulation results show that RPNN-EF is significantly faster than other RPNN-based models for one-step ahead forecasting and its forecasting performance is more significant than these models for multi-step ahead forecasting.

Multiresolution locally expanded HONN for handwritten numeral recognition

The detection of gaussian signals with unknown correlation coefficient, ρ s is considered. A strategy for designing high order neural networks (HONN) in composite hypothesis test is proposed. A HONN trained with ρ s varying uniformly in [0, 1] is considered to approximate the Average Likelihood Ratio (ALR). In order to compare the suitability of the approximation, a sub-optimal solution based on constrained generalized likelihood ratio is used. A study of the computational cost is carried out. Results show that a HONN is able to approximate the ALR with a low computational cost.

Business is a diversified field with general areas of specialisation such as accounting, taxation, stock market, and other financial analysis. Artificial Neural Networks (ANNs) have been widely used in ap- plications such as bankruptcy prediction, predicting costs, forecasting revenue, forecasting share prices and exchange rates, processing documents and many more. This chapter introduces an Adaptive Higher Order Neural Network (HONN) model and applies the adaptive model in business applications such as simulating and forecasting share prices. This adaptive HONN model offers significant advantages over traditional Standard ANN models such as much reduced network size, faster training, as well as much improved simulation and forecasting errors, due to their ability to better approximate complex, non-smooth, often discontinuous training data sets. The generalisation ability of this HONN model is explored and discussed.

This chapter introduces multi-polynomial higher order neural network models (MPHONN) with higher accuracy. Using Sun workstation, C++, and Motif, a MPHONN simulator has been built. Real-world data cannot always be modeled simply and simulated with high accuracy by a single polynomial function. Thus, ordinary higher order neural networks could fail to simulate complicated real-world data. But MPHONN model can simulate multi-polynomial functions and can produce results with improved accuracy through experiments. By using MPHONN for financial modeling and simulation, experimental results show that MPHONN can always have 0.5051% to 0.8661% more accuracy than ordinary higher order neural network models.

Traditional Hopfield networking has been widely used to solve combinatorial optimization problems. However, high order Hopfiled networks, as an expansion of traditional Hopfield networks, are seldom used to solve combinatorial optimization problems. In theory, compared with low order networks, high order networks have better properties, such as stronger approximations and faster convergence rates. In this chapter, the authors focus on how to use high order networks to model combinatorial optimization problems. Firstly, the high order discrete Hopfield Network is introduced, then the authors discuss how to find the high order inputs of a neuron. Finally, the construction method of energy function and the neural computing algorithm are presented. In this chapter, the N queens problem and the crossbar switch problem, which are NP-complete problems, are used as examples to illustrate how to model practical problems using high order neural networks. The authors also discuss the performance of high order networks for modeling the two combinatorial optimization problems.

In this paper, a new discrete time identification scheme for a singularly perturbed nonlinear system using recurrent high order multi-time scale neural network is presented. The high-order neural network (HONN) is known for its simple structure and powerful nonlinearity approximation property, which make it more suitable for modeling the singularly perturbed nonlinear systems than the multi-layer neural network [10]. An on-line identification scheme—optimal bounded ellipsoid (OBE) algorithm is developed for the recurrent high order neural network (RHONN) model. By adaptively changing the learning rate, the on-line identification scheme can achieve faster convergence compared to the other widely used learning schemes, such as backpropagation. Simulation results are presented to demonstrate the effectiveness of the proposed algorithm.