Semi-Supervised Learning

Graph-based semi-supervised learning (SSL) algorithms have been widely studied in the last few years. Most of these algorithms were designed from unconstrained optimization problems using a Laplacian regularizer term as smoothness functional in an attempt to reflect the intrinsic geometric structure of the data's marginal distribution. Although a number of recent research papers are still focusing on unconstrained methods for graph-based SSL, a recent statistical analysis showed that many of these algorithms may be unstable on transductive regression. Therefore, we focus on providing new constrained methods for graph-based SSL. We begin by analyzing the regularization framework of existing unconstrained methods. Then, we incorporate two normalization constraints into the optimization problem of three of these methods. We show that the proposed optimization problems have closed-form solution. By generalizing one of these constraints to any distribution, we provide generalized methods for constrained graph-based SSL. The proposed methods have a more flexible regularization framework than the corresponding unconstrained methods. More precisely, our methods can deal with any graph Laplacian and use higher order regularization, which is effective on general SSL taks. In order to show the effectiveness of the proposed methods, we provide comprehensive experimental analyses. Specifically, our experiments are subdivided into two parts. In the first part, we evaluate existing graph-based SSL algorithms on time series data to find their weaknesses. In the second part, we evaluate the proposed constrained methods against six state-of-the-art graph-based SSL algorithms on benchmark data sets. Since the widely used best case analysis may hide useful information concerning the SSL algorithms' performance with respect to parameter selection, we used recently proposed empirical evaluation models to evaluate our results. Our results show that our methods outperforms the competing methods on most parameter settings and graph construction methods. However, we found a few experimental settings in which our methods showed poor performance. In order to facilitate the reproduction of our results, the source codes, data sets, and experimental results are freely available.

In this article we present a semi‐supervised active learning algorithm for pattern discovery in information extraction from textual data. The patterns are reduced regular expressions composed of various characteristics of features useful in information extraction. Our major contribution is a semi‐supervised learning algorithm that extracts information from a set of examples labeled as relevant or irrelevant to a given attribute. The approach is semi‐supervised because it does not require precise labeling of the exact location of features in the training data. This significantly reduces the effort needed to develop a training set. An active learning algorithm is used to assist the semi‐supervised learning algorithm to further reduce the training set development effort. The active learning algorithm is seeded with a single positive example of a given attribute. The context of the seed is used to automatically identify candidates for additional positive examples of the given attribute. Candidate examples are manually pruned during the active learning phase, and our semi‐supervised learning algorithm automatically discovers reduced regular expressions for each attribute. We have successfully applied this learning technique in the extraction of textual features from police incident reports, university crime reports, and patents. The performance of our algorithm compares favorably with competitive extraction systems being used in criminal justice information systems.

Following the success of supervised learning, semi-supervised learning (SSL) is now becoming increasingly popular. SSL is a family of methods, which in addition to a labeled training set, also use a sizable collection of unlabeled data for fitting a model. Most of the recent successful SSL methods are based on pseudo-labeling approaches: letting confident model predictions act as training labels. While these methods have shown impressive results on many benchmark datasets, a drawback of this approach is that not all unlabeled data are used during training. We propose a new SSL algorithm, DoubleMatch, which combines the pseudo-labeling technique with a self-supervised loss, enabling the model to utilize all unlabeled data in the training process. We show that this method achieves state-of-the-art accuracies on multiple benchmark datasets while also reducing training times compared to existing SSL methods. Code is available at https://github.com/walline/doublematch.

Employing unlabeled data to improve the classification performance of SVM, and its application in audio event classification

A semi-supervised deep learning approach for cropped image detection

In many practical data mining applications, such as web categorization, key gene selection, etc., generally, unlabeled training examples are readily available, but labeled ones are fairly expensive to obtain. Therefore, in recent years, semi-supervised learning algorithms such as graph-based methods have attracted much attention. However, most of these traditional methods adopted a Gaussian function to calculate the edge weights of the graph. In this paper, a novel weight for semi-supervised graph-based methods is proposed. The new method adds the label information from problem into the target function, and adopts the geodesic distance rather than Euclidean distance as the measure of the difference between two data points when conducting the calculation. In addition, we also add class prior knowledge from problem into semi-supervised learning algorithm. Here we address the problem of learning with local and global consistency (LGC). It was found that the effect of class prior knowledge was probably different between under high label rate and low label rate. Furthermore, we integrate sparse representation (SR) in LGC algorithm. Experiments on a UCI data set show that our proposed method outperforms the original algorithms.KeywordsSemi-supervised learningClassification errorLabel informationGeodesic distanceClass prior knowledgesparse representation

Over the last decade, many pairwise-constraint-based metric learning algorithms have been developed to automatically learn application-specific metrics from data under similarity/dissimilarity data-pair constraints (weak labels). Nevertheless, these existing methods are designed for the centralized learning case, in which all the data and constraints are supposed to be gathered together in one source, and the algorithms utilize the whole data and constraints information during the learning process. However, in many real applications, large amounts of data (constraints) are dispersedly generated/stored in geographically distributed nodes over networks. Thus, it might be impractical to centralize the whole data information to one fusion node. Besides, in such cases, it is often hard to have every data pair labeled due to the huge data-pair amounts, resulting in numerous unlabeled data pairs. Given these situations, in this paper, we propose two types, namely, a diffusion type and an alternating-direction-method-of-multipliers type, of distributed semi-supervised metric learning frameworks, which make use of both labeled and unlabeled data pairs. The proposed frameworks can be easily used to extend centralized metric learning methods of different objective functions to distributed cases. In particular, we apply our frameworks on a well-behaved centralized semi-supervised metric learning method called SERAPH and yield two new distributed semi-supervised metric learning algorithms. Our simulation results show that the metrics learned by the proposed distributed algorithms are very close to that of the corresponding centralized method in most cases.

The amount of unlabeled linguistic data available to us is much larger and growing much faster than the amount of labeled data. Semi-supervised learning algorithms combine unlabeled data with a small labeled training set to train better models. This tutorial emphasizes practical applications of semisupervised learning; we treat semi-supervised learning methods as tools for building effective models from limited training data. An attendee will leave our tutorial with 1. A basic knowledge of the most common classes of semi-supervised learning algorithms and where they have been used in NLP before. 2. The ability to decide which class will be useful in her research. 3. Suggestions against potential pitfalls in semisupervised learning.

Wireless multimedia sensor networks (WMSN) have recently emerged as one of the most important technologies, driven by the powerful multimedia signal acquisition and processing abilities. Target classification is an important research issue addressed in WMSN, which has strict requirement in robustness, quickness and accuracy. This paper proposes a collaborative semi-supervised classifier learning algorithm to achieve durative online learning for support vector machine (SVM) based robust target classification. The proposed algorithm incrementally carries out the semi-supervised classifier learning process in hierarchical WMSN, with the collaboration of multiple sensor nodes in a hybrid computing paradigm. For decreasing the energy consumption and improving the performance, some metrics are introduced to evaluate the effectiveness of the samples in specific sensor nodes, and a sensor node selection strategy is also proposed to reduce the impact of inevitable missing detection and false detection. With the ant optimization routing, the learning process is implemented with the selected sensor nodes, which can decrease the energy consumption. Experimental results demonstrate that the collaborative hybrid semi-supervised classifier learning algorithm can effectively implement target classification in hierarchical WMSN. It has outstanding performance in terms of energy efficiency and time cost, which verifies the effectiveness of the sensor nodes selection and ant optimization routing.

Aiming at the problem of object-based image retrieval, a novel semi-supervised multi-instance learning (MIL) algorithm based on RS (rough set) attribute reduction and Transductive support vector machine (TSVM) has been presented---RSTSVM- MIL algorithm. This algorithm regards the whole image as a bag, and the low-level visual feature of the segmented regions as instances, in order to transform every bag to a single sample, RSTSVM-MIL first generates a collection of visual-word to construct projection space by K-means clustering method in the instance feature space, then a nonlinear mapping is defined using these visual-word and maps every bag to a point in the projection space, extracting all bags' projection feature, which converts MIL problem to a standard supervised learning problem. Finally, in order to improve the efficiency of classifier training and to use a large amount of unlabeled image together with the labeled image to build better classifiers, we use RS (rough set) method to reduce the redundant information in projection feature firstly, then semi-supervised learning algorithm TSVM be used to train classifier. Experimental results on the SIVAL image set show that solving MIL problems using semi-supervised learning is feasible, and the RSTSVM-MIL is competitive with other state-of-the-art MIL algorithms in object- based image retrieval problem.

Machine learning has became a popular method for intrusion detection due to self-adaption for changing situation. Limited to lack of high quality labeled instances, some researchers focused on semi-supervised learning to utilize unlabeled instances enhancing classification. But involving the unlabeled instances into learning process also introduces vulnerability: attackers can generate fake unlabeled instances to mislead the final classifier so that a few intrusions can not be detected. We show how attackers can influence the semi-supervised classifier by constructing unlabeled instances in this paper. And a possible defence method which based on active learning is proposed. Experiments show that the misleading attack can reduce the accuracy of the semi-supervised learning method and the presented defense method against the misleading attack can obtain higher accuracy than the original semi-supervised learner under the proposed attack.

The existing Synthetic Aperture Radar (SAR) image target detection methods based on convolutional neural networks (CNNs) have achieved remarkable performance, but these methods require a large number of target-level labeled training samples to train the network. Moreover, some clutter is very similar to targets in SAR images with complex scenes, making the target detection task very difficult. Therefore, a SAR target detection network based on a semi-supervised learning and attention mechanism is proposed in this paper. Since the image-level label simply marks whether the image contains the target of interest or not, which is easier to be labeled than the target-level label, the proposed method uses a small number of target-level labeled training samples and a large number of image-level labeled training samples to train the network with a semi-supervised learning algorithm. The proposed network consists of a detection branch and a scene recognition branch with a feature extraction module and an attention module shared between these two branches. The feature extraction module can extract the deep features of the input SAR images, and the attention module can guide the network to focus on the target of interest while suppressing the clutter. During the semi-supervised learning process, the target-level labeled training samples will pass through the detection branch, while the image-level labeled training samples will pass through the scene recognition branch. During the test process, considering the help of global scene information in SAR images for detection, a novel coarse-to-fine detection procedure is proposed. After the coarse scene recognition determining whether the input SAR image contains the target of interest or not, the fine target detection is performed on the image that may contain the target. The experimental results based on the measured SAR dataset demonstrate that the proposed method can achieve better performance than the existing methods.

Hardware of Distributed WSNs is limited, which turns into a big challenge. In recent years, there are some pattern-based query strategies have been proposed. But compared to the methods based on machine learning, these methods need high energy consumption in the calculation of distance. In this paper, we propose a pattern-based query strategy based on semi-supervised machine learning in wireless sensor network. The strategy uses historical time series data streams for o†ine machine learning. First of all, we flnd out the flxed length subsequences and screen out the suitable data sequences as training dataset and sample dataset. And then, because labeled data is less in practical application, we do label propagation with graph-based semi-supervised machine learning algorithm, Local and Global Consistency (LGC), to increase the number of labeled data sequences. Next, we use the supervised learning algorithm Support Vector Machine (SVM) to train a classifler. Finally, for a new data sequence, we can directly get its category through the classifler. Experimental results show that the performance of our strategy is almost better than other pattern-based query strategies on classiflcation accuracy and energy consumption.

Semi-supervised matrixized least squares support vector machine

State-of-the-art data analysis tools have to deal with high-dimensional data. Fortunately, the inherent dimensionality of data is often much smaller, as it has an internal structure limiting its degrees of freedom. In most cases, this structure can be approximated using a graph, i.e, a set of nodes connected by edges. Based on this idea, graphs have been largely used in semi-supervised and unsupervised learning. The canonical assumption when incorporating a graph prior is to assume that the signal is smooth with respect to the graph, i.e, that connected pieces of data have similar values. Nevertheless, this hypothesis is insufficient to characterize more complex relations between the data and its structure, the graph. To this end, the field of Graph Signal Processing (GSP) extends the graph smoothness hypothesis to a more general assumption. Its key idea is to think of the signal as a sum of harmonic modes, which are obtained from the eigen-decomposition of the Laplacian, an operator representing the second order derivative. Thus, the essence of GSP resides in the Fourier transform defined by the projection on the Laplacian eigenvectors basis, which allows us to filter graph signals. However, GSP suffers from a counterintuitive irregularity. Contrarily to the classical Fourier basis, the graph Laplacian eigenvectors are not spreading uniformly on the vertex set and can be highly concentrated in a region of the graph. This non-uniformity results in a lack of any intuitive translation operator and consequently poses challenges to generalize some of the classical signal processing theory, e.g., stationarity, uncertainty principles, or sampling. In this thesis, we answer these difficulties by accepting that each node is unique and should be treated specially according to its surrounding structure. This is achieved using an operator called localization that shifts a kernel defined in the spectral domain around a specific node while adapting to the local graph structure. In the first part of the thesis, we focus on harmonic analysis in GSP. We start by making use of the norm of localized kernels to capture the spectral local features of the graph vertices. To illustrate the relevance of this analysis we then introduce structural clustering, an algorithm that groups nodes according to the role they play in the graph. Then, we bound the vertex-frequency concentration of graph filter banks atoms. Because of the irregular structure of graphs, we construct a local uncertainty principle that is, again, driven by the norm of a localized kernel. In the second part of the thesis, we tackle GSP problems from a machine learning perspective. We use the localization operator to extend the notion of stationarity to graph signals. It consists of a graph signal probabilistic framework allowing us to optimally solve inverse problems. Finally, we show how the graph total variation can be used in semi-supervised learning and prove its consistency with respect to the underlying graph manifold.