- Research Article
13
- 10.1016/j.asoc.2021.107455
From WASD to BLS with application to pattern classification
- Apr 30, 2021
- Applied Soft Computing
- Mei Liu + 4 more +4
From WASD to BLS with application to pattern classification
Neural networks have evolved into one of the most critical tools in the field of artificial intelligence. As a kind of shallow feedforward neural network, the broad learning system (BLS) uses a training process based on random and pseudoinverse methods, and it does not need to go through a complete training cycle to obtain new parameters when adding nodes. Instead, it performs rapid update iterations on the basis of existing parameters through a series of dynamic update algorithms, which enables BLS to combine high efficiency and accuracy flexibly. The training strategy of BLS is completely different from the existing mainstream neural network training strategy based on the gradient descent algorithm, and the superiority of the former has been proven in many experiments. This article applies an ingenious method of pseudoinversion to the weight updating process in BLS and employs it as an alternative strategy for the dynamic update algorithms in the original BLS. Theoretical analyses and numerical experiments demonstrate the efficiency and effectiveness of BLS aided with this method. The research presented in this article can be regarded as an extended study of the BLS theory, providing an innovative idea and direction for future research on BLS.
From WASD to BLS with application to pattern classification
From WASD to BLS with application to pattern classification
Stream label distribution learning processing via broad learning system
Stream label distribution learning processing via broad learning system
Advancing broad learning through structured feature generation
• Tackles randomness redundancy in broad learning systems with structured design. • Improves accuracy compared to classic broad learning systems in classification and regression tasks. • Addresses the lack of robustness in random neural architectures when faced with noise and scarce data. • Structured network design produces interpretable feature maps, enabling transparent AI decisions. Deep neural networks achieve strong performance in big data scenarios, while requiring extensive iterative parameter optimization, making them inefficient and suboptimal in scarce data scenarios. Broad Learning System (BLS) has gained popularity as an efficient, effective, and incremental learning model. BLS relies on independent and identically distributed random feature generation. Although efficient, the literature has shown that this approach can lead to suboptimal and redundant representations. This paper introduces Structured BLS (SBLS), a novel reinterpretation of BLS components. SBLS enhances latent features by incorporating a structured random basis, which provides a beneficial inductive bias that promotes neuronal specialization to learn specific patterns in the data while reducing the redundancy issue of the classic BLS. Experimental results in various classification and regression datasets demonstrate that SBLS outperforms BLS in terms of performance, robustness to noise, and interpretability, while remaining efficient and easy to deploy. Our findings emphasize the need for focused feature generation through random weights in neural networks and reservoir computing. In fact, we are transitioning from a chaotic to a controlled exploration of patterns. Moreover, we illustrate how our approach can incorporate task-specific knowledge into neuron behavior by design. SBLS has practical implications for real-world applications that involve data scarcity. By refining the way randomness is exploited in neural networks, our work challenges the conventional wisdom that improved performance requires deeper architectures or complex optimization strategies. Instead, we show that intelligent feature generation can unlock significant gains at minimal additional cost. 1
Read moreRCBLS: An Outlier-Robust Broad Learning Framework with Compact Structure
Recently, the broad learning system (BLS) has been widely developed in virtue of its excellent performance and high-computational efficiency. However, two deficiencies still exist in BLS and preclude its deployment in real applications. First, the standard BLS performs poorly in outlier environments because the least squares loss function it uses to train the network is sensitive to outliers. Second, the model structure of BLS is likely to be redundant since the hidden nodes in it are randomly generated. To address the above two issues, a new robust and compact BLS (RCBLS), based on M-estimator and sparsity regularization, is proposed in this paper. The RCBLS develops from the BLS model and maintains its excellent characteristics, but replaces the conventional least squares learning criterion with an M-estimator-based loss function that is less sensitive to outliers, in order to suppress the incorrect feedback of the model to outlier samples, and hence enhance its robustness in the presence of outliers. Meanwhile, the RCBLS imposes the sparsity-promoting l2,1 -norm regularization instead of the common l2-norm regularization for model reduction. With the help of the row sparsity of l2,1-norm regularization, the unnecessary hidden nodes in RCBLS can be effectively picked out and removed from the network, thereby resulting in a more compact network. The theoretical analyses on outlier robustness, structural compactness and computational complexity of the proposed RCBLS model are provided. Finally, the validity of the RCBLS is verified by regression, time series prediction and image classification tasks. The experimental results demonstrate that the proposed RCBLS has stronger anti-outlier ability and more compact network structure than BLS and other representative algorithms.
Read moreTwo-Dimensional Broad Learning System for Data Analytic
Recently a novel learning algorithm called the broad learning system (BLS) has been established by the theory of pseudo-inverse and compressed sensing technology. As an alternative to deep learning, BLS has gained extensive attention in data analytics. Unlike the existing deep structures based learning tools (such as deep neural networks), it is able to rapidly achieve incremental learning, and does not require tedious retraining to remodel the system. However, given image data modeling tasks, one-dimensional BLS requires a conventional vectorization operation, the dimension of input will be quite large for high resolution image. This operation may result in huge computational complexity, thereby increasing the time of model construction. To resolve this issue, a two-dimensional version of BLS, namely 2D-BLS, is first proposed to fast construct randomized learner models by using matrix inputs. Specifically, the proposed method employs left and right projecting vectors to replace the usual high dimensional input weight. Moreover, the theoretical analyses on the superiority of 2D-BLS against BLS are presented in terms of the algorithm complexity and the difference of random parameter space. Empirical results based on MNIST dataset, NORB 3D object image recognition dataset, ORL dataset, YaleB dataset and Handwritten Digit dataset demonstrate that the proposed 2D-BLS outperforms the original BLS, some extensions of BLS and some 2-D machine learning methods in terms of the efficiency of model construction. This result also shows good potential of 2D-BLS for image data analytics.
Read moreAnalysis and Variants of Broad Learning System
The broad learning system (BLS) is designed based on the technology of compressed sensing and pseudo-inverse theory, and consists of feature nodes and enhancement nodes, has been proposed recently. Compared with the popular deep learning structures, such as deep neural networks, BLS has the ability of rapid incremental learning and can remodel the system without the usual tedious retraining process. However, given that BLS is still in its infancy, it still needs analysis, improvements, and verification. In this article, we first analyze the principle of fast incremental learning ability of BLS in depth. Second, in order to provide an in-depth analysis of the BLS structure, according to the novel structure design concept of deep neural networks, we present four brand-new BLS variant networks and their incremental realizations. Third, based on our analysis of the effect of feature nodes and enhancement nodes, a new BLS structure with a semantic feature extraction layer has been proposed, which is called SFEBLS. The experimental results show that SFEBLS and its variants can increase the accuracy rate on the NORB dataset 6.18%, Fashion-MNIST dataset by 3.15%, ORL data by 5.00%, street view house number dataset by 12.88%, and CIFAR-10 dataset by 18.42%, respectively, and the four brand-new BLS variant networks also obviously outperform the original BLS.
Read moreA Fault Aware Broad Learning System for Concurrent Network Failure Situations
The broad learning system (BLS) framework gives an efficient solution for training flat-structured feedforward networks and flat structured deep neural networks. However, the classical BLS model and other variants focus on the faultless situation only, where enhancement nodes, feature mapped nodes, and output weights of a BLS network are assumed to be realized in a perfect condition. When a trained BLS network suffers from coexistence of weight/node failures, the trained network has a greatly degradation in its performance if a countermeasure is not taken. In order to reduce the effect of weight/node failures on the BLS network's performance, this paper proposes an objective function for enhancing the fault aware performance of BLS networks. The objective function contains a fault aware regularizer term which handles the weight/node failures. A learning algorithm is then derived based on the objective function. The simulation results show that the performance of the proposed fault aware BLS (FABLS) algorithm is superior to the classical BLS and two state-of-the-arts BLS algorithms, namely correntropy criterion BLS (CBLS) and weighted BLS (WBLS).
Read moreFrequency Principle in Broad Learning System.
Deep neural networks have achieved breakthrough improvement in various application fields. Nevertheless, they usually suffer from a time-consuming training process because of the complicated structures of neural networks with a huge number of parameters. As an alternative, a fast and efficient discriminative broad learning system (BLS) is proposed, which takes the advantages of flat structure and incremental learning. The BLS has achieved outstanding performance in classification and regression problems. However, the previous studies ignored the reason why the BLS can generalize well. In this article, we focus on the interpretation from the viewpoint of the frequency domain. We discover the existence of the frequency principle in BLS, i.e., the BLS preferentially captures low-frequency components quickly and then fits the high frequencies during the incremental process of adding feature nodes and enhancement nodes. The frequency principle may be of great inspiration for expanding the application of BLS.
Read moreLearning broad learning system with controllable sparsity through L0 regularization
As a novel neural network with efficient learning capacity, broad learning system (BLS) has achieved remarkable success in various regression and classification problems. Due to the broad expansion of nodes, however, BLS is known to have many redundant parameters and nodes, which will increase the memory and computation cost and is adverse to its deployment on equipment with limited resources. To optimize the number of neurons and parameters of BLS and then find the optimal sparse model under a given resource budget, in this paper, we introduce to train BLS through L0 regularization. The regularization constraint term of the BLS objective function is replaced by the L0 regularization method, and the normalized hard threshold iterative method is used to optimize the output weight. More concretely, the size of the model is fixed by controlling the number of output weights under given the resource size, and then parameters and nodes in the network are evaluated and selected from the node set in the training to obtain a BLS with controllable sparsity (CSBLS). Experiments on various data sets demonstrate the effectiveness of our proposed method.
Read moreEEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks and Broad Learning System
In recent years, electroencephalogram (EEG) e-motion recognition has been becoming an emerging field in artificial intelligence area, which can reflect the relation between emotional states and brain activity. In this paper, we designed a novel architecture, i.e., broad dynamical graph learning system (BDGLS), to deal with EEG signals. By integrating the advantage of dynamical graph convolution neural networks (DGCNN) and broad learning system (BLS), BDGLS has the ability of extracting features on non-Euclidean domain and randomly generating nodes to find the desired connection weights simultaneously. We evaluated our system on SJTU emotion EEG dataset (SEED), and used differential entropy (DE) features as input data. In the experiments, BDGLS achieved the best result, compared with the state-of-the-art methods, e.g., support vector machine (SVM), deep belief networks (DBN), graph convolutional neural networks (DCNN) and DGCNN. Especially the performance on all-frequency band of DE features, BDGLS reached the highest average recognition accuracy of 93.66% with the standard deviation of 6.11%. The result demonstrated the excellent classification ability of BDGLS in EEG emotion recognition.
Read moreStacked Broad Learning System: From Incremental Flatted Structure to Deep Model
The broad learning system (BLS) has been proved to be effective and efficient lately. In this article, several deep variants of BLS are reviewed, and a new adaptive incremental structure, Stacked BLS, is proposed. The proposed model is a novel incremental stacking of BLS. This invariant inherits the efficiency and effectiveness of BLS that the structure and weights of lower layers of BLS are fixed when the new blocks are added. The incremental stacking algorithm computes not only the connection weights between the newly stacking blocks but also the connection weights of the enhancement nodes within the BLS block. The Stacked BLS is considered as the increment of “layers” and “neurons” dynamically during the training for multilayer neural networks. The proposed architecture along with the training algorithms that utilizes the residual characteristic is very versatile in comparison with traditional fixed architecture. Finally, experimental results on UCI datasets, MNIST dataset, NORB dataset, CIFAR-10 dataset, SVHN dataset, and CIFAR-100 dataset indicate that the proposed method outperforms the selected state-of-the-art methods on both accuracy and training speed, such as deep residual networks. The results also imply that the proposed structure could highly reduce the number of nodes and the training time of the original BLS in the classification task of some datasets.
Read moreResearch Review for Broad Learning System: Algorithms, Theory, and Applications.
In recent years, the appearance of the broad learning system (BLS) is poised to revolutionize conventional artificial intelligence methods. It represents a step toward building more efficient and effective machine-learning methods that can be extended to a broader range of necessary research fields. In this survey, we provide a comprehensive overview of the BLS in data mining and neural networks for the first time, focusing on summarizing various BLS methods from the aspects of its algorithms, theories, applications, and future open research questions. First, we introduce the basic pattern of BLS manifestation, the universal approximation capability, and essence from the theoretical perspective. Furthermore, we focus on BLS's various improvements based on the current state of the theoretical research, which further improves its flexibility, stability, and accuracy under general or specific conditions, including classification, regression, semisupervised, and unsupervised tasks. Due to its remarkable efficiency, impressive generalization performance, and easy extendibility, BLS has been applied in different domains. Next, we illustrate BLS's practical advances, such as computer vision, biomedical engineering, control, and natural language processing. Finally, the future open research problems and promising directions for BLSs are pointed out.
Read moreA Classification for Electronic Nose Based on Broad Learning System
The odor of citrus juice changes during the storing process. We use an electronic nose (E-nose) to detect the volatile odors released by citrus juice and use the detect results to classify citrus juices from different storage periods. In this article, a novel classifier of E-nose, namely broad learning system (BLS) is introduced. BLS is different from traditional classifier. It has a simple network model, which can greatly reduce the training time of the model. BLS can effectively combine feature extraction and classification recognition to make the model more efficient. We apply BLS to the analysis of valencia citrus juice data. The experimental results show that BLS can effectively identify the current stage of the stored valencia citrus juice. Compared with traditional classifier such as radical basis function neural network (RBFNN) and linear discriminant analysis (LDA), the results show that BLS has better performance for the storage period classification of valencia citrus juice.
Read moreFacial Beauty Prediction Combining Dual-Branch Feature Fusion With a Stacked Broad Learning System
Facial beauty prediction (FBP) is a key computer vision task that uses algorithms to assess facial attractiveness. Current models rely on single feature extraction, such as using a single convolutional neural network to extract local feature, failing to capture other potentially more important information contained within facial data and limiting feature diversity. Dual-branch feature fusion complements key features from multiple models, reducing reliance on single features. A stacked broad learning system (BLS) increases network depth with residual connections, addressing the feature insensitivity of traditional BLS. Therefore, this paper combines dual-branch feature fusion with a stacked BLS for FBP. First, facial features are extracted by the ConvNeXtBase and MobileNetV3Large models through transfer learning. Second, a squeeze-and-excitation (SE) network attention-based connection layer dynamically adjusts feature weights, emphasizing important features and reducing irrelevant ones. Then, facial features are input into the BLS network, and the outputs are horizontally concatenated to form fused features. Finally, the fused features are reinput into the BLS to obtain the final FBP result. This model is referred to as DB-BLS (dual branch broad learning system). Building upon DB-BLS, this paper introduces a stacked BLS module to replace the BLS module. Stacked BLS enhancing the network’s ability to perceive features at different levels. This model is named SDB-BLS (stacked dual branch broad learning system). Experiments show that DB-BLS and SDB-BLS outperform traditional methods, respectively achieving accuracies of 76.86% and 78.78% on the SCUT-FBP5500 dataset, with potential applications in image classification and object detection.
Read moreRich Feature Combination for Cost-Based Broad Learning System
Broad learning system (BLS) is an emerging learning algorithm for the connectionist models, which have enjoyed much popularity on many applications. As an alternative approach of learning in deep structure, the BLS develops an incremental learning neural network that can be modeled in a flexible way, and becomes a promising technique in the field of knowledge discovery and data engineering. To further improve the performance of BLS, our focus is to investigate these algorithms which can enhance the BLS. On one hand, from the viewpoint of feature engineering, unsupervised group-wise encoding is conducted for feature extraction, and broadly fused feature representation is used to improve the ability of BLS, in terms of the learning and reusing multi-level features. On the other hand, for imbalanced learning from disproportionate size of categories instances, a cost-sensitive BLS framework is proposed in this paper, which aims to minimize the total misclassifying cost in classification learning. Finally, we conduct extensive experiments on a wide range of datasets (e.g., computer vision and bug reports) to demonstrate the effectiveness of the proposed BLS framework.
Read more