- Research Article
5
- 10.1016/j.neucom.2013.10.033
Semi-supervised learning via sparse model
- Dec 01, 2013
- Neurocomputing
- Yu Wang + 4 more +4
Semi-supervised learning via sparse model
Semi-supervised learning (SSL) methods provide a powerful tool for utilizing abundant unlabeled data to strengthen standard supervised learning. Traditional graph-based SSL methods prevail in classical SSL problems for their intuitional implementation and effective performance. However, they encounter troubles when applying to image classification followed by modern deep learning, since the diffusion algorithms face the curse of dimensionality. In this study, we propose a simple and efficient SSL method, combining a graph-based SSL paradigm with differential privacy. We aim at developing coherent latent feature space of deep neural networks so that the diffusion algorithm in the latent space can give more precise predictions for unlabeled data. Our approach achieves state-of-the-art performance on the Cifar10, Cifar100, and Mini-imagenet benchmark datasets and obtains an error rate of 18.56% on Cifar10 using only 1% of all labels. Furthermore, our approach inherits the benefits of graph-based SSL methods with a simple training process and can be easily combined with any network architecture.
Loading PDF
Semi-supervised learning via sparse model
Semi-supervised learning via sparse model
Graph-Based Semi-Supervised Learning: A Comprehensive Review.
Semi-supervised learning (SSL) has tremendous value in practice due to the utilization of both labeled and unlabelled data. An essential class of SSL methods, referred to as graph-based semi-supervised learning (GSSL) methods in the literature, is to first represent each sample as a node in an affinity graph, and then, the label information of unlabeled samples can be inferred based on the structure of the constructed graph. GSSL methods have demonstrated their advantages in various domains due to their uniqueness of structure, the universality of applications, and their scalability to large-scale data. Focusing on GSSL methods only, this work aims to provide both researchers and practitioners with a solid and systematic understanding of relevant advances as well as the underlying connections among them. The concentration on one class of SSL makes this article distinct from recent surveys that cover a more general and broader picture of SSL methods yet often neglect the fundamental understanding of GSSL methods. In particular, a significant contribution of this article lies in a newly generalized taxonomy for GSSL under the unified framework, with the most up-to-date references and valuable resources such as codes, datasets, and applications. Furthermore, we present several potential research directions as future work with our insights into this rapidly growing field.
Read moreAutomatic image annotation by semi-supervised manifold kernel density estimation
Automatic image annotation by semi-supervised manifold kernel density estimation
Image/Video Semantic Analysis by Semi-Supervised Learning
The insufficiency of labeled training samples is a major obstacle in automatic semantic analysis of large scale image/video database. Semi-supervised learning, which attempts to learn from both labeled and unlabeled data, is a promising approach to tackle this problem. As a major family of semi-supervised learning, graph-based methods have attracted more and more recent research. In this chapter, a brief introduction is given on popular semi-supervised learning methods, especially the graph-based methods, as well as their applications in the area of image annotation, video annotation, and image retrieval. It is well known that the pair-wise similarity is an essential factor in graph propagation based semisupervised learning methods. A novel graph-based semi-supervised learning method, named Structure- Sensitive Anisotropic Manifold Ranking (SSAniMR), is derived from a PDE based anisotropic diffusion framework. Instead of using Euclidean distance only, SSAniMR further takes local structural difference into account to more accurately measure pair-wise similarity. Finally some future directions of using semi-supervised learning to analyze the multimedia content are discussed.
Read moreGraph-based semi-supervised learning: A review
Graph-based semi-supervised learning: A review
Video semantic analysis based on structure-sensitive anisotropic manifold ranking
Video semantic analysis based on structure-sensitive anisotropic manifold ranking
Semi-Supervised Learning with Density-Sensitive Manifold graph
The key problem of Graph-Based Semi-Supervised Learning (GBSSL) methods is how to construct the graph structure under some assumptions. While distance information among graph nodes is investigated well for graph construction, the density information is not given enough attention. In this paper, we propose a novel GBSSL method, named Density-Sensitive Manifold Learning (DSML), which introduces density distribution into graph construction by calculating a new propagation coefficient matrix. The experimental results show that DSML scheme performs better than traditional GBSSL methods. More importantly, the new propagation coefficient matrix can be easily introduced into traditional GBSSL methods to improve their performance, which is also validated in the experiments.
Read moreParticle Competition and Cooperation to Prevent Error Propagation from Mislabeled Data in Semi-supervised Learning
Semi-supervised learning is applied to classification problems where only a small portion of the data items is labeled. In these cases, the reliability of the labels is a crucial factor, because mislabeled items may propagate wrong labels to a large portion or even the entire data set. This paper aims to address this problem by presenting a graph-based (network-based) semi-supervised learning method, specifically designed to handle data sets with mislabeled samples. The method uses teams of walking particles, with competitive and cooperative behavior, for label propagation in the network constructed from the input data set. The proposed model is nature-inspired and it incorporates some features to make it robust to a considerable amount of mislabeled data items. Computer simulations show the performance of the method in the presence of different percentage of mislabeled data, in networks of different sizes and average node degree. Importantly, these simulations reveals the existence of the critical points of the mislabeled subset size, below which the network is free of wrong label contamination, but above which the mislabeled samples start to propagate their labels to the rest of the network. Moreover, numerical comparisons have been made among the proposed method and other representative graph-based semi-supervised learning methods using both artificial and real-world data sets. Interestingly, the proposed method has increasing better performance than the others as the percentage of mislabeled samples is getting larger.
Read moreInstance selection method for improving graph-based semi-supervised learning
Graph-based semi-supervised learning is an important semi-supervised learning paradigm. Although graph-based semi-supervised learning methods have been shown to be helpful in various situations, they may adversely affect performance when using unlabeled data. In this paper, we propose a new graph-based semi-supervised learning method based on instance selection in order to reduce the chances of performance degeneration. Our basic idea is that given a set of unlabeled instances, it is not the best approach to exploit all the unlabeled instances; instead, we should exploit the unlabeled instances that are highly likely to help improve the performance, while not taking into account the ones with high risk. We develop both transductive and inductive variants of our method. Experiments on a broad range of data sets show that the chances of performance degeneration of our proposed method are much smaller than those of many state-of-the-art graph-based semi-supervised learning methods.
Read moreSemi-Supervised Concept Learning by Concept-Cognitive Learning and Concept Space
In human concept learning, people can naturally combine a handful of labeled data with abundant unlabeled data when they make classification decisions, which is also known as semi-supervised learning (SSL) in machine learning. Especially, human concept learning not only is a static process in human cognition but also can vary gradually with dynamic environments. Nevertheless, the classical SSL algorithms must be redesigned to accommodate newly input data. In this sense, concept-cognitive learning may be a good choice, as it can implement dynamic processes by imitating human cognitive processes. Meanwhile, numerous SSL methods were designed based on the feature vector information of instances, while ignoring concept structural information that is a very important process in human knowledge organization. Based on this idea, a novel SSL method, named semi-supervised concept learning method (S2CL), is proposed for dynamic SSL by employing concept spaces, in which knowledge is represented by hierarchical concept structures. Moreover, to make full use of the global and local conceptual information, we further propose an extended version of S2CL (namely, <inline-formula><tex-math notation="LaTeX">$\text{S2CL}^{\alpha }$</tex-math></inline-formula> ) for concept learning. More specifically, to effectively exploit the unlabeled data, this paper first shows some new related theories for S2CL (or <inline-formula><tex-math notation="LaTeX">$\text{S2CL}^{\alpha }$</tex-math></inline-formula> ) based on a regular formal decision context; then a novel SSL framework is designed, and its corresponding algorithm is developed. Finally, we conduct some experiments on various datasets to demonstrate the effectiveness of our methods, which include concept classification and incremental learning under a large quantity of unlabeled data.
Read moreSpeaker Attribution with Voice Profiles by Graph-Based Semi-Supervised Learning
Speaker attribution is required in many real-world applications, such as\nmeeting transcription, where speaker identity is assigned to each utterance\naccording to speaker voice profiles. In this paper, we propose to solve the\nspeaker attribution problem by using graph-based semi-supervised learning\nmethods. A graph of speech segments is built for each session, on which\nsegments from voice profiles are represented by labeled nodes while segments\nfrom test utterances are unlabeled nodes. The weight of edges between nodes is\nevaluated by the similarities between the pretrained speaker embeddings of\nspeech segments. Speaker attribution then becomes a semi-supervised learning\nproblem on graphs, on which two graph-based methods are applied: label\npropagation (LP) and graph neural networks (GNNs). The proposed approaches are\nable to utilize the structural information of the graph to improve speaker\nattribution performance. Experimental results on real meeting data show that\nthe graph based approaches reduce speaker attribution error by up to 68%\ncompared to a baseline speaker identification approach that processes each\nutterance independently.\n
Read moreALG: Adaptive low-rank graph regularization for scalable semi-supervised and unsupervised learning
ALG: Adaptive low-rank graph regularization for scalable semi-supervised and unsupervised learning
Capped l_2,1-Norm Regularized Dictionary Coding for Scalable Semi-Supervised Learning
During the past decade, graph-based semi-supervised learning has become one of the most important research areas in machine learning and artificial intelligence community. In this paper, we propose a Capped l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> -Norm Regularized Dictionary Learning to construct the graph for semi-supervised learning (SSL). The new sparse coding is robust to the outliers and insensitive to noise. Then, the developed graph construction can better characterize the geometrical structure of data mainfold and be utilized for efficient graph based SSL. the proposed SSL methods can also be easily extended to deal with out-of-sample data. Simulation results show that the proposed method can achieve better performance compared with other state-of-the-art graph based SSL methods.
Read moreMulti granularity based label propagation with active learning for semi-supervised classification
Multi granularity based label propagation with active learning for semi-supervised classification
Anisotropic Manifold Ranking for Video Annotation
Graph-based semi-supervised learning (SSL) has attracted lots of interests in machine learning community as well as many application areas including video annotation recently. However, one of the two basic assumptions, structure assumption, which is an essential point of graph-based SSL, is not embedded into the pairwise similarity measure. Accordingly, we propose a novel graph-based SSL method for video annotation, named anisotropic manifold ranking (AniMR), based on a structure-related similarity measure. This method takes the influence of the density difference between samples into account to improve the pairwise similarity. Furthermore, we will show that AniMR can also be deduced from partial differential equation (PDE) based anisotropic diffusion. It demonstrates that the label propagation in AniMR is anisotropic, which is intrinsically different from the isotropic label propagation process in general graph-based SSL methods. Experiments conducted on the TRECVID data set show this approach outperforms ordinary graph-based SSL methods and is effective for video semantic annotation.
Read more