Multi-instance Multi-label Learning by Extreme Learning Machine

Multi instance multi label learning is a framework in which objects are represented as bags of instances and labels are provided at the bag level. Instance annotation is the problem of assigning labels to the instances in a bag given only the bag label. Recently, OR-ed logistic regression (OR-LR) model and an EM based inference method have been proposed for instance annotation. Due to the linear nature of the logistic regression function, OR-LR performance on linearly inseparable data is limited. This paper addresses this problem by proposing a regularized kernel-based extension to the OR-LR framework. Experiments show that the kernel-based OR-LR algorithm achieves a significant improvement in classification accuracy over the linear OR-LR from 3% to 9% on audio bird song and image annotation datasets and two synthetic datasets.

As a powerful learning tool, Extreme Learning Machine (ELM) shows its merits in classification, regression and clustering by offering both high prediction accuracy and high learning speed. Among numerous ELM varieties, multiple kernel ELM draws intensive attention from researchers because it can leverage information from multiple heterogeneous sources, which is a common scenario in big data era. Despite remarkable efforts for supervised multiple kernel ELM, few publications have addressed the unsupervised case, which is more critical yet challenging for tackling realistic problem. In this paper, we address this problem by proposing a two-stage unsupervised multiple kernel extreme learning machine, which is suitable for fast multiple-view clustering. This approach learns the cluster and kernel combination weights alternatively. At the first stage, it generates cluster label based on a given combined kernel. Then, at the second stage, the kernel combination weights are learned by distance label based extreme learning machine based on the label generated at the previous stage. Experimental results on both synthetic and real data sets demonstrate its outstanding performance in term of both accuracy and learning speed.

Automated annotation of protein function is challenging. As the number of sequenced genomes rapidly grows, the vast majority of proteins can only be annotated computationally. Nature often brings several domains together to form multi-domain and multi-functional proteins with a vast number of possibilities, and each domain may fulfill its own function independently or in a concerted manner with its neighbors. Thus, it is evident that the protein function prediction problem is naturally and inherently Multi-Instance Multi-Label (MIML) learning tasks. Based on the state-of-the-art MIML algorithm MIMLNN, we propose a novel ensemble MIML learning framework EnMIMLNN and design three algorithms for this task by combining the advantage of three kinds of Hausdorff distance metrics. Experiments on seven real-world organisms covering the biological three-domain system, i.e., archaea, bacteria, and eukaryote, show that the EnMIMLNN algorithms are superior to most state-of-the-art MIML and Multi-Label learning algorithms.

Nature often brings several domains together to form multidomain and multifunctional proteins with a vast number of possibilities. In our previous study, we disclosed that the protein function prediction problem is naturally and inherently Multi-Instance Multilabel (MIML) learning tasks. Automated protein function prediction is typically implemented under the assumption that the functions of labeled proteins are complete; that is, there are no missing labels. In contrast, in practice just a subset of the functions of a protein are known, and whether this protein has other functions is unknown. It is evident that protein function prediction tasks suffer from weak-label problem; thus protein function prediction with incomplete annotation matches well with the MIML with weak-label learning framework. In this paper, we have applied the state-of-the-art MIML with weak-label learning algorithm MIMLwel for predicting protein functions in two typical real-world electricigens organisms which have been widely used in microbial fuel cells (MFCs) researches. Our experimental results validate the effectiveness of MIMLwel algorithm in predicting protein functions with incomplete annotation.

In this paper, we propose the MIML (Multi- Instance Multi-Label learning) framework which is associated with multiple class labels for Image Annotation. Compared to traditional learning frameworks, the MIML framework is more convenient and natural for representing complicated objects which have multiple semantic meanings. To learn from MIML examples we have taken a survey on MIML Boost, MIMLSVM, D-MIMLSVM, InsDif and SubCod algorithms. MIML Boost and MIML SVM are based on a simple degeneration strategy. Experiments based on this algorithm shows that solving problems involving complicated objects with multiple semantic meanings in the MIML framework can lead to good performance. As the degeneration process may lose information, we have considered D-MimlSvm algorithm which tackles MIML problems directly in a regularization framework. InsDif and SubCod algorithms works by transforming single-instances into the MIML representation for learning, while SubCod works by transforming single-label examples into the MIML representation for learning. We have compared the results of all the algorithms and have identified that InsDif framework leads to good performance rates.

In this paper, we seek to solve the problem of remote sensing image classification using Multi Instance Multi Label Learning (MIML) framework, where each image contain multiple regions (instances) corresponds to multiple objects (labels). MIML framework is more efficient than traditional learning framework for complicated objects such as satellite images. Existing MIML algorithms such as MIMLboost, MIMLsvm and MIMLfast have been found useful in scene classification and can provide multiple labels for each instance in complicated objects. The proposed approach was performed in two steps: In the first step, we process the dataset using segmentation and feature extraction. Images are ambiguous because they contain numerous objects so we consider that each image is a bag and various blocks in the image are instances. In the second step, we apply MIML to classify each block of images. According to the experimental results, the proposed method outperforms the state-of-the-art methods.

The single-hidden-layer feedforward neural networks (SLFNs) are frequently used in machine learning due to their ability which can form boundaries with arbitrary shapes if the activation function of hidden units is chosen properly. Most learning algorithms for the neural networks based on gradient descent are still slow because of the many learning steps. Recently, a learning algorithm called extreme learning machine (ELM) has been proposed for training SLFNs to overcome this problem. It randomly chooses the input weights and hidden-layer biases, and analytically determines the output weights by the matrix inverse operation. This algorithm can achieve good generalization performance with high learning speed in many applications. However, this algorithm often requires a large number of hidden units and takes long time for classification of new observations. In this paper, a new approach for training SLFNs called least-squares extreme learning machine (LS-ELM) is proposed. Unlike the gradient descent-based algorithms and the ELM, our approach analytically determines the input weights, hidden-layer biases and output weights based on linear models. For training with a large number of input patterns, an online training scheme with sub-blocks of the training set is also introduced. Experimental results for real applications show that our proposed algorithm offers high classification accuracy with a smaller number of hidden units and extremely high speed in both learning and testing.

This paper presents a new optimal controller design for the sensorless variable speed wind turbine (SVSWT) based on the extreme learning machine (ELM) and the adaptive particle swarm optimisation (APSO) algorithms. The two main objectives of this command are to maximize the conversion of wind energy below the rated wind speed and to maintain the safety of the wind turbine system (WT) by minimizing stress on the drive train shafts. The proposed technique is based on the efficiency of the ELM for single hidden layer feed forward neural networks (SLFN) combined to sliding mode control (SMC) to respectively, improve the used model and stabilize the operation of the WT. ELM algorithm with high learning speed is used to approximate the nonlinear unmodelled dynamics while SMC is used to compensate the external disturbances and modelling errors. APSO algorithm is introduced to adapt and optimize the gain of the SMC. The efficiency of the proposed method is illustrated in simulations by the comparison with traditional SMC.

In many real world applications, the concerned objects are with multiple labels, and can be represented as a bag of instances. Multi-instance Multi-label (MIML) learning provides a framework for handling such task and has exhibited excellent performance in various domains. In a MIML setting, the feature representation of instances usually has big impact on the final performance; inspired by the recent deep learning studies, in this paper, we propose the DeepMIML network which exploits deep neural network formation to generate instance representation for MIML. The sub-concept learning component of the DeepMIML structure reserves the instance-label relation discovery ability of MIML algorithms; that is, it can automatically locating the key input patterns that trigger the labels. The effectiveness of DeepMIML network is validated by experiments on various domains of data.

Extreme Learning Machine (ELM) is a learning algorithm proposed recently to train single hidden layer feed forward networks (SLFN). It has many attractive properties that include better generalization performance and very fast learning. ELM starts by assigning random values to the input weights and hidden biases and then in one step it determines the output weights using Moore-Penrose generalized inverse. Despite the aforementioned advantages, ELM performance might be affected by the random initialization of weights and biases or by the large generated network which might contain unnecessary number of neurons. In order to increase the generalization performance and to produce more compact networks, a hybrid model that combines ELM with competitive swarm optimizer (CSO) is proposed in this paper. The proposed model (CSONN-ELM) optimizes the weights and biases and dynamically determines the most appropriate number of neurons. To evaluate the effectiveness of the CSONN-ELM, it is experimented using 23 benchmark datasets, and compared to a set of static rules extracted from literature that are used to determine the number of neurons of SLFN. Moreover, it is compared to two dynamic methods that are used to enhance the performance of ELM, that are Optimally pruned ELM (OP-ELM) and metaheuristic based ELMs (Particle Swarm Optimization-ELM and Differential Evolution-ELM). The obtained results show that the proposed method enhances the generalization performance of ELM and overcomes the static and dynamic methods.

In multi-instance learning, each example is represented by a bag of instances while associated with a binary label. Under standard multi-instance learning settings, one example is labeled as a positive bag if at least one of its instances is positive. Otherwise, it is labeled as a negative bag. Although based on the above assumption, standard multi-instance learning has achieved much success in solving diverse learning tasks, there are still many real-world problems where this assumption may not necessarily hold. Therefore, researchers aimed to expand the underlying assumption of standard multi-instance learning where two frameworks of generalized multi-instance learning have been proposed. In this paper, the problem definition, learning algorithms and also experimental data sets related to either generalized multi-instance learning framework are briefly reviewed.

The extreme learning machine (ELM), which was originally proposed for “generalized” single-hidden layer feedforward neural networks (SLFNs), provides efficient unified learning solutions for the applications of regression and classification. It presents competitive accuracy with superb efficiency in many applications. However, due to its single-layer architecture, feature selection using ELM may not be effective for natural signals. To address this issue, this paper proposes a new ELM-based multi-layer learning framework for dimension reduction. The novelties of this work are as follows: (1) Unlike the existing multi-layer ELM methods in which all hidden nodes are generated randomly, in this paper some hidden layers are calculated by replacement technologies. By doing so, more important information can be exploited for feature learning, which lead to a better generalization performance. (2) Unlike the existing multi-layer ELM methods which only work for sparse representation, the proposed method is designed for dimension reduction. Experimental results on several classification datsets show that, compared to other feature selection methods, the proposed method performs competitively or much better than other feature selection methods with fast learning speed.

Extreme Learning Machine (ELM), which is a Well-known supervised learning algorithm, provides a high generalization performance and a fast learning technique. However, ELM cannot handle the class imbalance. Recently, the weighted ELM was developed to solve this issue. The weighted ELM performance still faced with finding the proper boundary between positive class and negative class. In this paper, we proposed GWO-weighted ELM that is a combination of the weighted ELM integrated with the Gray wolf optimizer (GWO). The main contribution of the paper is the regularization parameter is optimized via the GWO algorithm. The experiments were conducted on 16 real imbalanced datasets. We compared the proposed GWO-weighted ELM with the original weighted ELM. The experimental results showed that the GWO-weighted ELM with Wl is superior to the weighted ELM and the GWO-weighted ELM with W2.

This paper addresses the analysis of the problem of combining Infinite Kernel Learning (IKL) approach and Extreme Learning Machine (ELM) structure. ELM represents a novel and promising alternative to Neural Networks, for its simplicity in implementation and high efficiency, especially concerning convergence and generalization performance. A currently underdeveloped topic concerning ELM implementation is given by the optimization process of base kernels: choosing different kernel combinations may lead to very dissimilar performance results. An innovative ELM approach using a combination of multiple kernels has been proposed in Liu et al. As a change of paradigm, we are interested in using an infinite set of base kernels, defining in this way an original ELM based algorithm called Infinite Kernel Extreme Learning Machine (IK-ELM). About that, a novel 3-step algorithm combining IKL and ELM is proposed. Finally, a brief analysis about further possible directions is discussed.

Extreme learning machine (ELM) has gained increasing interests from various research fields recently. Researchers have proposed various extensions to improve its stability, sparsity, and generalization performance. In this paper, we propose a robust and sparse ELM to exploit <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L_{21} $ </tex-math></inline-formula> -norm minimization of both loss function and regularization (LR21-ELM). Our <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L_{21} $ </tex-math></inline-formula> -norm-based loss function can diminish the undue influence of noises and outliers of data points compared with the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L_{2} $ </tex-math></inline-formula> -norm based loss function and make the learned ELM model more robust and stable. The powerful structural sparse-inducing <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L_{21} $ </tex-math></inline-formula> -norm regularization is integrated into the ELM objective function to eliminate the potential redundant neurons of ELM adaptively and reduce the complexity of the learning model. We introduce an effective iterative optimization algorithm to solve the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L_{21} $ </tex-math></inline-formula> -norm minimization problem. Empirical tests on a number of benchmark datasets indicate that our proposed algorithm can generate a more compact, robust, and discriminative model compared with the original ELM algorithm.