Combining multiple representations and classifiers for pen-based handwritten digit recognition

Mixture-of-Experts (MoE) large language models (LLM) have memory requirements that often exceed the GPU memory capacity, requiring costly parameter movement from secondary memories to the GPU for expert computation. In this work, we present Mixture of Near-Data Experts (MoNDE), a near-data computing solution that efficiently enables MoE LLM inference. MoNDE reduces the volume of MoE parameter movement by transferring only the hot experts to the GPU, while computing the remaining cold experts inside the host memory device. By replacing the transfers of massive expert parameters with the ones of small activations, MoNDE enables far more communication-efficient MoE inference, thereby resulting in substantial speedups over the existing parameter of-floading frameworks for both encoder and decoder operations.

This paper investigates the use of artificial neural networks (ANN) in risk estimation of asset returns. Istanbul Stock Exchange (ISE) index (XU100) is studied with a mixture of experts ANN architecture using daily data over a 12-year period. Results are compared to feed-forward neural networks, multilayer perceptron (MLP) and radial basis function (RBF) networks and recurrent neural networks (RNN). They are also compared to widely accepted Exponential Generalized Autoregressive Conditional Heteroskedasticity (EGARCH) volatility model. These results suggest that mixture of experts (MoE) have the strength to capture the volatility in index return series and prepares a valuable basis for financial decision making.KeywordsRoot Mean Square ErrorMean Square ErrorRadial Basis FunctionRecurrent Neural NetworkHide UnitThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

We applied multilayer perceptron (MLP) and radial basis function (RBF) neural networks using data from two water quality monitoring stations at the Karaj Dam in Iran. Input data were calcium ions (Ca2+), magnesium ions (Mg2+), sodium ions (Na+), chloride ions (Cl−), sulfate ( SO 42−), and pH, and the output data were total dissolved solids (TDS). An MLP with one hidden layer containing eight neurons was selected for the upstream water quality station using normalized input data. We developed a second MLP neural network for the downstream station with one hidden layer containing 10 neurons in the hidden layer using normalized input data. Considering applying normalized input data and one hidden layer, the coefficient of determination (R2) and index of agreement (IA) between the observed and the predicted data for the upstream and downstream monitoring stations using the MLP neural networks were 0.985, 0.84, 0.99, and 0.92, respectively. The RBF neural network with 100 neurons in its hidden layer reached the minimum errors between the observed and the predicted results in upstream and downstream stations. The R2 between observed and predicted data for upstream and downstream monitoring stations for the RBF was 0.999 and 0.998, respectively. Data normalization improved the performance of the MLP neural networks. Sensitivity analysis indicated that magnesium is the most effective water quality parameter for predicting TDS, and sulfate is the second most effective water quality parameter affecting TDS prediction at the Karaj Dam.

To generalize the Fisher Discriminant Analysis (FDA) algorithm to the case of discriminant functions belonging to a nonlinear, finite dimensional function space F (Nonlinear FDA or NFDA), it is sufficient to expand the input data by computing the output of a basis of F when applied to it [1,2,3,4]. The solution to NFDA can then be found like in the linear case by solving a generalized eigenvalue problem on the between- and within-classes covariance matrices (see e.g. [5]). The goal of NFDA is to find linear projections of the expanded data (i.e., nonlinear transformations of the original data) that minimize the variance within a class and maximize the variance between different classes. Such a representation is of course ideal to perform classification. The application of NFDA to pattern recognition is particularly appealing, because for a given input signal and a fixed function space it has no parameters and it is easy to implement and apply. Moreover, given C classes only C - 1 projections are relevant [5]. As a consequence, the feature space is very small and the algorithm has low memory requirements and high speed during recognition. Here we apply NFDA to a handwritten digit recognition problem using the MNIST database, a standard and freely available set of 70,000 handwritten digits (28 × 28 pixels large), divided into a training set (60,000 digits) and a test set (10,000 digits). Several established pattern recognition methods have been applied to this database by Le Cun et al. [6]. Their paper provides a standard reference work to benchmark new algorithms. We perform NFDA on spaces of polynomials of a given degree d, whose corresponding basis functions include all monomials up to order d in all input variables. It is clear that the problem quickly becomes intractable because of the high memory requirements. For this reason, the input dimensionality is first reduced by principal component analysis. On the preprocessed data we then apply NFDA by expanding the training patterns in the polynomial space and solving the linear FDA eigenvalue problem. As mentioned above, since we have 10 classes we only need to compute the first 9 eigenvectors. Since the within-class variance is minimized, the patterns belonging to different classes tend to cluster in the feature space when projected on the eigenvectors. For this reason we classify the digits with a simple method such as Gaussian classifiers. We perform simulations with polynomials of degree 2 to 5. With polynomials of degree 2 the explosion in the dimensionality of the expanded space with increasing number of input dimensions is relatively restricted, so that it is possible to use up to 140 dimensions. With higher order polynomials one has to rely on a smaller number of input dimensions, but since the function space gets larger and includes new nonlinearities, one obtains a remarkable improvement in performance. The best performance is achieved with polynomials of degree 3 and 35 input dimensions, with an error rate of 1.5% on test data. This error rate is comparable to but does not outperform that of the most elaborate algorithms (Table 1). The performance of NFDA is however remarkable considering the simplicity of the method and the fact that it has no a priori knowledge on the problem, in contrast for example to the LeNet-5 algorithm [6] which has been designed specifically for handwritten character recognition. In addition, for recognition, NFDA has to store and compute only 9 functions and has thus small memory requirements and a high recognition speed. It is also possible to formulate NFDA using the kernel trick, in which case one can in principle use function spaces of infinite dimensionality [1,2,3,4]. However, the limiting factor in that formulation is the number of training patterns, which makes it not realistic for this application. The performance of NFDA could be further improved using for example a more problem-specific preprocessing of the patterns (e.g., by increasing the size of the training set with new patterns generated by artificial distortion of the original one), boosting techniques, or mixture of experts with other algorithms [5,6].

Breast cancer is the most common kind of cancer, which is the cause of death among the women worldwide. There is evidence that shows that the early detection and treatment can increase the survival rate of patients who suffered this disease. Therefore, this paper proposes an automatic breast cancer diagnosis technique using a genetic algorithm for simultaneous feature selection and parameter optimization of an Multi Layer Perceptron (MLP) neural network. The aim of this paper is to propose a hybrid classification algorithm based on Multi-stage Weights Adjustment in the MLP (MWAMLP) neural network in two parts to improve the breast cancer diagnosis. In the first part, the three classifiers are trained simultaneously on the learning dataset. The output of the first part classifier together with the learning dataset is placed in a new dataset. This dataset uses a hybrid classifier method to model the mapping between the outputs of each ordinary classifier of the first part with real output labels. The proposed algorithm is implemented with three different variations of the backpropagation (BP) technique, namely the Levenberg–Marquardt, resilient BP and gradient descent with momentum for fine tuning of the weight of MLP neural network and their performances are compared. Interestingly, one of the proposed algorithms titled MWAMLP-RP produces the best and on average, 99.35% and 98.74% correct classification, respectively, on the Wisconsin Breast Cancer Database dataset, which is comparable with the obtained results from the methods titled GP-DLNN, GAANN and CAFS and other works found in the literature.

In this paper, a multi-layered perceptron (MLP) neural network (NN) is proposed to nullify the effects of the measured data errors associated with phasor measurement units (PMUs). With this MLP NN, power system voltage stability is estimated by calculating voltage collapse indices (VCIs). Several VCIs are compared with two evaluation criteria to determine those appropriate indices that should be trained using NN as the target output. In other words, the data obtained from the PMU are exploited as the MLP NN inputs, and the exact informative VCIs are considered as its output. The capability of the proposed method is demonstrated by applying it to the New England 39-bus test system. The obtained results confirm the validity of the proposed technique.

Hydraulic calculations of water distribution systems are currently performed using computer programs. In addition to the basic calculation procedure, modules responsible for evaluating the obtained calculation results are introduced more and more often into the programs. This article presents the results of research on artificial neural networks with a radial base function (RBF) and a multilayer perceptron (MLP), aimed at determining whether they can be used to model the relationship between the variables describing the computational section of the water distribution system and the diameter of the water pipe. The classification capabilities of the RBF and MLP networks were analyzed according to the number of neurons in the hidden layer of the network. A comparative analysis of RBF networks with multilayer perceptron (MLP) networks was performed. The results showed that the MLP networks have much better classification properties and are better suited for the task of assessing the selected diameters of the water pipes.

Complexity Analysis of Multilayer Perceptron Neural Network Embedded into a Wireless Sensor Network

Applies modular neural network models to classify sleep-waking states in infants. The performances of three connectionist models are compared: (a) a multilayer perceptron (MLP), (b) a mixture of experts (ME) and (c) a fuzzy ganglionar lattice (FGL). We propose a new methodology for enhancing neural classifiers based on input variable selection and confusion error analysis using expert criteria. The ME model was more robust than the MLP and FGL models in the presence of inconsistent or noisy data. Input variable selection and confusion error analysis using expert criteria led to parsimonious models with less parameters and better classification rates.

Bidikmisi scholarships are tuition assistance for prospective students who are economically disadvantaged but have the good academic potential to study at State Universities in Indonesia . With the amount Bidikmisi applicants, there is a need to use a system that can speed up the selection process and avoid human errors. The purpose of this study is to develop a classification system for Bidikmisi scholarship applicants at Universities in Indonesia. This study uses a multilayer perceptron (MLP) neural network to build a classification model. Because MLP has some hidden layers, MLP has advantages to predict or analyze complex problems. The methodology in this study is data collection, data preprocessing, data encoding, data modeling, and evaluation. The results of the classification of bidikmisi scholarship applicants are divided into two classes, namely accepted and rejected. The multilayer perceptron method generates an accuracy value of 72.81%. The Multilayer perceptron is compared with other classification methods, namely Naïve Bayes and K-Nearest Neighbors (KNN). The results of the classification using the Naïve Bayes method obtained an accuracy value of 79.47%, while the accuracy value using KNN was 78.33%. Evaluation using MLP resulted in precision, recall, and F1-measure values of 68.36%, 72.81%, and 66.33%. the results of the evaluation using Naïve Bayes were 75.58%, 79.47%, and 71.24%, while the KNN values were 71.90%, 78.33%, and 72.53%. In order to the conclusion in this research is the MLP method can generate good performance for the classification of bidikmisi scholarship applicants. However, Naïve Bayes’ performance is better than MLP for the classification of bidikmisi scholarship applicants.

For evaluating the accuracy and capability of MLP (Multi Layer Perceptron) and RBF (Radial Basis Function) classifier algorithms; projection profile features for the character images are extracted and are merged with the binarization features after preprocessing every character image, and the combined features thus obtained are used to train both the classifiers (MLP and RBF Network) selected for performing character recognition task. Simulation studies are examined extensively and the proposed hybrid features are found to deliver the recognition accuracy of 93.46% when used with RBF Network as a classifier which is better than the 88.76% accuracy obtained when used with MLP Network as a classifier.

Estimating impervious surfaces from medium spatial resolution imagery using the self-organizing map and multi-layer perceptron neural networks

Neural networks are a soft computing technique with wide application in signal processing as well as system and process modeling. In the present study, multilayer perceptron (MLP) neural networks were employed to process multidimensional signals generated in metal machining operations (including three-dimensional cutting force signals and three-dimensional cutting vibration signals) and to establish a model for predicting the machined surface roughness. This paper describes in detail our methods of multidimensional signal processing and modeling with MLP neural networks. The MLP neural network model developed in the present study fills an important research gap by taking into account the critical effect of tool-edge radius in machining. As compared to regression models, the MLP neural network model developed in the present study has significantly higher accuracy in predicting the machined surface roughness.

We applied multilayer perceptron (MLP) and radial basis function (RBF) neural network in upstream and downstream water quality stations of the Karaj Reservoir in Iran. For both neural networks, inputs were pH, turbidity, temperature, chlorophyll-a, biochemical oxygen demand (BOD) and nitrate, and the output was dissolved oxygen (DO). We used an MLP neural network with two hidden layers, for upstream station 15 and 33 neurons in the first and second layers respectively, and for the downstream station, 16 and 21 neurons in the first and second hidden layer were used which had minimum amount of errors. For learning process 6-fold cross validation were applied to avoid over fitting. The best results acquired from RBF model, in which the mean bias error (MBE) and root mean squared error (RMSE) were 0.063 and 0.10 for the upstream station. The MBE and RSME were 0.0126 and 0.099 for the downstream station. The coefficient of determination (<TEX>$R^2$</TEX>) between the observed data and the predicted data for upstream and downstream stations in the MLP was 0.801 and 0.904, respectively, and in the RBF network were 0.962 and 0.97, respectively. The MLP neural network had acceptable results; however, the results of RBF network were more accurate. A sensitivity analysis for the MLP neural network indicated that temperature was the first parameter, pH the second and nitrate was the last factor affecting the prediction of DO concentrations. The results proved the workability and accuracy of the RBF model in the prediction of the DO.

Local learning techniques associated with multilayer perceptron (MLP) networks typically employ receptive fields as an integral part of the network. However, data representation schemes that employ multiple, overlapping receptive fields to preprocess network inputs can be another source of local learning in MLP networks. Earlier work has shown that ensemble encoding, a distributed data representation scheme, promotes local learning and can accelerate learning in MLP networks. We demonstrate that networks using ensemble encoding display an enhanced capacity for incremental learning.