A Multiple Hidden Layers Extreme Learning Machine Method and Its Application

Deep neural networks have made significant achievements in representation learning of traditionally man-made features, especially in terms of complex objects. Over the decades, this learning process has attracted thousands of researchers and has been widely used in the speech, visual, and text recognition fields. One deep network multi-layer extreme learning machine (ML-ELM) achieves a good performance in representation learning while inheriting the advantages of faster learning and the approximating capability of the extreme learning machine (ELM). However, as with most deep networks, the ML-ELM’s algorithmic performance largely depends on the probability distribution of the training data. In this paper, we propose an improved ML-ELM made via using the local significant regions at the input end to enhance the contributions of these regions according to the idea of the selective attention mechanism. To avoid involving and exploring the complex principle of the attention system and to focus on the clarification of our local regional enhancement idea, the paper only selects two typical attention regions. One is the geometric central region, which is normally the important region to attract human attention due to the focal attention mechanism. The other is the task-driven interest region, with facial recognition as an example. The comprehensive experiments are done on the three public datasets of MNIST, NORB, and ORL. The comparison experiment results demonstrate that our proposed region-enhanced ML-ELM (RE-ML-ELM) achieves performance increases in important feature learning by utilizing the apriori knowledge of attention and has a higher recognition rate than that of the normal ML-ELM and the basic ELM. Moreover, it benefits from the non-iterative parameter training method of other ELMs, and our proposed algorithm outperforms most state-of-the-art deep networks such as deep belief network(DBN), in the aspects of training efficiency. Furthermore, because of the deep structure with fewer hidden nodes at each layer, our proposed RE-ML-ELM achieves a comparable training efficiency to that of the ML-ELM but has a higher training speed with the basic ELM, which is normally the width single network that has more hidden nodes to obtain the similar recognition accuracy with the deep networks. Based on our idea of combining the apriori knowledge of the human selective attention system with the data learning, our proposed region-enhanced ML-ELM increases the image classification performance. We believe that the idea of intentionally combining psychological knowledge with the most algorithms based on data-driven learning has the potential to improve their cognitive computing ability.

Determination of bubble point pressure & oil formation volume factor of crude oils applying multiple hidden layers extreme learning machine algorithms

Multiple faults are easily confused with single faults. In order to identify multiple faults more accurately, a highly efficient learning method is proposed based on a double parallel two-hidden-layer extreme learning machine, called DPT-ELM. The DPT-ELM method is a variant of an extreme learning machine (ELM). There are some issues with ELM. First, achieving a high accuracy requires too many hidden nodes; second, the direct connection between the input layer and the output layer is ignored. Accordingly, to deal with the above-mentioned problems, DPT-ELM extends the single-hidden-layer ELM to a two-hidden-layer ELM, which can achieve a desired performance with fewer hidden nodes. In addition, a direct connection is built between the input layer and the output layer. Since the input layer weights and the thresholds of the two hidden layers are determined randomly, this simplifies the improved model and shortens the calculation time. Additionally, to improve the signal to noise ratio (SNR), an adaptive waveform decomposition (AWD) algorithm is used to denoise the vibration signal. Then, the denoised signal is used to extract the eigenvalues by the time-domain and frequency-domain methods. Finally, the eigenvalues are input to the DPT-ELM classifier. In this paper, two groups of rolling bearing data at different speeds, which were collected from a real experimental platform, are used to test the method. Each set of data includes three single fault states, two complex fault states and a healthy state. The experimental results demonstrate that the DPT-ELM method achieves fast learning speed and a high accuracy. Moreover, based on 10-fold cross-validation, it proves to be an effective method to improve the accuracy with fewer hidden nodes.

Traffic flow modeling has a pivotal role within Intelligent Transportation Systems (ITSs), holding vital importance in alleviating traffic congestion and decreasing carbon emissions. Due to the presence of variability and nonlinear attributes in traffic flow, developing an effective and resilient model for predicting traffic flow poses a significant challenge. Precisely predicting traffic flow is not merely a feasible issue; it also poses significant difficulties to the researchers involved in this field. This study proposes a hybrid predictive model to forecast traffic flow. The proposed model effectively merges the strengths of the Sparrow Search algorithm (SSA) and Multi-layer Extreme Learning Machine (ML-ELM) model, enhancing prediction accuracy. SSA optimization technique is applied to optimize the initial weights and bias parameters for ML-ELM model. ELM approach is a machine learning approach that employs a single hidden layer to address various tasks. However, in situations where more complex problems are encountered, ML-ELM extends this concept by incorporating multiple hidden layers to enhance its capabilities and address challenges more effectively. Finally, SSA technique is utilized to achieve the optimal tuning of hyperparameters in the context of ML-ELM model to improve the prediction accuracy. Compared to the other selected models, the proposed model outperforms them in terms of performance metrics, including Root Mean Square Errors (RMSE), Mean Absolute Errors (MAE), Mean Absolute Percentage Errors (MAPE) and Correlation Coefficients (r), indicating that it is appropriate for this prediction task.

Recently, multilayer extreme learning machine with subnetwork nodes and cost sensitive learning was applied to representation learning and classification. However, the existing ML-ELM methods suffer from several drawbacks: 1) manual tuning for the number of hidden nodes in each layer will influent the training speed of the model, the best optimal determine approach for the number of hidden nodes is still for probing; 2) random projection of the input weight and bias in each layer still suboptimal and the network structure hard to reach best generalization performance. Inspired by the Subnetwork nodes and cost sensitive methods, using multiple hidden output matrix, a novel ELM method which called Multilayer Incremental Hybrid Cost Sensitive Extreme Learning Machine with Multiple Hidden Output Matrix and Subnetwork Hidden Nodes is proposed in this paper. Whose contributions are 1) a hybrid hidden nodes optimization methods is given which based on ant clone method and multiple grey wolf optimization method; 2) a multiple hidden layer output matrices are being using through weighted calculation of different output matrices which can optimize the network architecture effectively; 3) a novel ELM structure is proposed by merging the merits of Cost Sensitive Extreme Learning Machine and Subnetwork Hidden Nodes, while the regularized parameter $C$ is not sensitive to the generalization performance. Extensive experiments results show that the proposed method achieves better performance than existing state-of-art ELM methods.

Recently, multilayer extreme learning machine (ELM) algorithms have been extensively studied for hierarchical abstract representation learning in the ELM community. In this paper, we investigate the specific combination of $$L_{21}$$ -norm based loss function and regularization to improve the robustness and the sparsity of multilayer ELM. As we all known, the mean square error (MSE) cost function (or squared $$L_{2}$$ -norm cost function) is commonly used as optimization cost function for ELM, but it is sensitive to outliers and impulsive noises that are pervasive in real-world data. Our $$L_{21}$$ -norm loss function can lessen the harmful influence caused by noises and outliers and enhance robustness and stability of the learned model. Additionally, the row sparse inducing $$L_{21}$$ -norm regularization can learn the most-relevant sparse representation and reduce the intrinsic complexity of the learning model. We propose a specific combination of $$L_{21}$$ -norm loss function and regularization ELM auto-encoder (LR21-ELM-AE), and then stack LR21-ELM-AE hierarchically to construct the hierarchical extreme learning machine (H-LR21-ELM). Experiments conducted on several well-known benchmark datasets are presented, the results show that the proposed H-LR21-ELM can generate a more robust, more discriminative and sparser model compared with the other state-of-the-art multilayer ELM algorithms.

Human activity recognition (HAR) is a main research field of context-aware computing; the performance of HAR mainly depends on the feature extraction method and classification algorithm. Extreme learning machine (ELM) is a single hidden layer neural network, which has better classification and generalization ability. However, ELM is not suitable for feature extraction. Deep learning is a hot research field as it can automatically extract significant features from raw data. In this paper, we propose an approach: an ELM-based deep model, which combined convolutional neural network (CNN), multilayer ELM (ML-ELM) as feature extractor, and used kernel ELM (KELM) as classifier. We used CNN and ML-ELM to extract significant features, and used KELM to achieve stable performance. The performance of proposed approach is validated on two public HAR datasets, and the experimental results show that the proposed approach is effective.

Information security plays critical roles in modern society. However, traditional security measures like passwords, tokens and personal ID numbers only provide limited protection. Inspired by the fast training speed of extreme learning machine (ELM) and the promising feature extraction capability of extreme learning machine auto-encoder (ELM-AE), this paper proposes a keystroke dynamics identification method for intelligent keyboard (IKB) based on multi-layer extreme learning machine (ML-ELM). The IKB, as first demonstrated by Wang's group, is a self-powered, non-mechanical perforated keyboard that converts mechanical stimuli applied to the keyboard into electrical signals without the needing an external power source. ML-ELM is a multilayer neural network stacking on top of ELM-AE. Our major contribution is to develop an accurate and efficient keystroke dynamics identification method based on ML-ELM. One significant advantage of the proposed method is that it does not rely on manual feature extraction and selection. In other words, the raw current signals obtained by IKB are directly the input to the network. The network structure is determined by grid search, which minimizes the human involvement. The other one is that the whole training process of the model does not require fine tuning, which makes the training process significantly faster than that of existing deep networks. Experimental results demonstrate that the proposed method has excellent timeliness whereas obtaining high identification accuracy. The proposed method has great potential in applications to computer or network access control, online payment, and cyber security.

Fully connected multi layer neural networks such as Deep Boltzmann Machines (DBM) performs better than fully connected single layer neural networks in image classification tasks and has a smaller number of hidden layer neurons than Extreme Learning Machine (ELM) based fully connected multi layer neural networks such as Multi Layer ELM (MLELM) and Hierarchical ELM (H-ELM) However, ML-ELM and H-ELM has a smaller training time than DBM. This paper introduces a fully connected multi layer neural network referred to as Multi Layer Multi Objective Extreme Learning Machine (MLMO-ELM) which uses a multi objective formulation to pass the label and non-linear information in order to learn a network model which has a similar number of hidden layer parameters as DBM and smaller training time than DBM. The experimental results show that MLMO-ELM outperforms DBM, ML-ELM and H-ELM on OCR and NORB datasets.

A novel recognition method was put forward to identify the producing areas of the flue-cured tobacco leaves rapidly and non-destructively by using a near-infrared (NIR) spectrometer and a multi-layer-extreme learning machine (ML-ELM) algorithm. In contrast to traditional linear discriminant analysis (LDA) and extreme learning machine (ELM) algorithms, the accuracy, sensitivity and specificity were the highest for the proposed ML-ELM algorithm. The ML-ELM models for different producing areas of Yunnan tobacco leaves had the best generalization ability and prediction results. Besides, the above three algorithms were also identified by using the chemical index data. The experimental results indicated that the NIR spectroscopy technology together with ML-ELM algorithm achieved the best prediction performance both using the NIR spectral data and chemical index data. It indicates that the combination of NIR and ML-ELM can recognize different producing areas of Yunnan tobacco leaves rapidly, accurately, and non-destructively.

The learning speed of the feed forward neural network takes a lot of time to be trained which is a major drawback in their applications since the past decades. The key reasons behind may be due to the slow gradient-based learning algorithms which are extensively used to train the neural networks or due to the parameters in the networks which are tuned iteratively using some learning algorithms. Thus, in order to eradicate the above pitfalls, a new learning algorithm was proposed known as Extreme Learning Machines (ELM). This algorithm tries to compute Hidden-layer-output matrix that is made of randomly assigned input layer and hidden layer weights and randomly assigned biases. Unlike the other feedforward networks, ELM has the access of the whole training dataset before going into the computation part. Here, we have devised a new two-layer-feedforward network (TFFN) for ELM in a new manner with randomly assigning the weights and biases in both the hidden layers, which then calculates the output-hidden layer weights using the Moore-Penrose generalized inverse. TFFN doesn't restricts the algorithm to fix the number of hidden neurons that the algorithm should have. Rather it searches the space which gives an optimized result in the neurons combination in both the hidden layers. This algorithm provides a good generalization capability than the parent Extreme Learning Machines at an extremely fast learning speed. Here, we have experimented the algorithm on various types of datasets and various popular algorithm to find the performances and report a comparison.

A data-attribute-space-oriented double parallel (DASODP) structure for enhancing extreme learning machine: Applications to regression datasets

In this paper, we have proposed a new algorithm to train neural network, called Class-Constrained Extreme Learning Machine (C\(^{2}\)ELM), which is based on Extreme Learning Machine (ELM). In C\(^{2}\)ELM, we use class information to constrain different parts of connection weights between input layer and hidden layer using Extreme Learning Machine Auto Encoder (ELM-AE). In this way, we add class information to the connection weights and make the features in the hidden layer which are learned from input space be more discriminative than other methods based on ELM. Meanwhile, C\(^{2}\)ELM can retain the advantages of ELM. The experiments shown that C\(^{2}\)ELM is effective and efficient and can achieve a higher performance in contrast to other ELM based methods.

In Indonesia, there are more than 2.039 species of herbal medicinal plants, which sometimes have similarities and make it difficult to identify the type of herbal plant. The purpose of this study is to facilitate the identification of herbal plant species by comparing the performance of the Extreme Learning Machine (ELM) and Multi-Support Vector Machine (Multi-SVM) methods. The ELM method was created to overcome the weaknesses of feedforward artificial neural networks, especially in terms of learning speed, while the Multi-SVM method is an advanced development of the SVM method. The stages of this research begin with image input which is through previous data acquisition, data preprocessing, and then the identification with ELM and Multi-SVM methods. Based on the simulations that have been carried out, the average accuracy on training data for the ELM method is 93%, while the Multi-SVM method is 44%. Also, the average accuracy on testing data for the ELM method is 85%, while the Multi-SVM method is 40%.

The shallow seawater depth inversion based on remote sensing technology is important for water depth detection, which is of considerable significance to marine engineering, shipping, and marine military security. In this study, we took the Taiping Island and its adjacent waters in the South China Sea as a test bed and developed a water depth inversion model on the basis of extreme learning machine (ELM) and extreme learning machine optimized by genetic algorithm (GA-ELM). In GA-ELM, the input weights and the hidden layer biases were optimized by genetic algorithm. The two models allowed the evaluation of nonlinear relationships between the reflectance of high-resolution imagery from WorldView-2 and actual water depth obtained from the S-57 sea chart. The eight bands of the high-resolution image and the actual water depth were used as the input layer and the output layer, and the sigmoid function was introduced as activation function. Finally, the model accuracy was evaluated by using mean relative error (MRE), root mean square error (RMSE), mean absolute error (MAE), coefficient of determination (R2), and the regression analysis between the retrieved water depth and the actual data. The simulation results showed that the two models had better stability than the second-order polynomial regression, BP neural network, and RBF neural network. Furthermore, GA-ELM had a more compact network structure and better generalization ability than ELM. Thus, we concluded that GA-ELM had higher precision and could achieve a better inversion result in the experimental area.