Learning Bayesian network classifiers for facial expression recognition both labeled and unlabeled data

Active Learning is a machine learning and data mining technique that selects the most informative samples for labeling and uses them as training data; it is especially useful when there are large amount of unlabeled data and labeling them is expensive. Recently, batch-mode active learning, where a set of samples are selected concurrently for labeling, based on their collective merit, has attracted a lot of attention. The objective of batch-mode active learning is to select a set of informative samples so that a classifier learned on these samples has good generalization performance on the unlabeled data. Most of the existing batch-mode active learning methodologies try to achieve this by selecting samples based on certain criteria. In this article we propose a novel criterion which achieves good generalization performance of a classifier by specifically selecting a set of query samples that minimize the difference in distribution between the labeled and the unlabeled data, after annotation. We explicitly measure this difference based on all candidate subsets of the unlabeled data and select the best subset. The proposed objective is an NP-hard integer programming optimization problem. We provide two optimization techniques to solve this problem. In the first one, the problem is transformed into a convex quadratic programming problem and in the second method the problem is transformed into a linear programming problem. Our empirical studies using publicly available UCI datasets and two biomedical image databases demonstrate the effectiveness of the proposed approach in comparison with the state-of-the-art batch-mode active learning methods. We also present two extensions of the proposed approach, which incorporate uncertainty of the predicted labels of the unlabeled data and transfer learning in the proposed formulation. In addition, we present a joint optimization framework for performing both transfer and active learning simultaneously unlike the existing approaches of learning in two separate stages, that is, typically, transfer learning followed by active learning. We specifically minimize a common objective of reducing distribution difference between the domain adapted source, the queried and labeled samples and the rest of the unlabeled target domain data. Our empirical studies on two biomedical image databases and on a publicly available 20 Newsgroups dataset show that incorporation of uncertainty information and transfer learning further improves the performance of the proposed active learning based classifier. Our empirical studies also show that the proposed transfer-active method based on the joint optimization framework performs significantly better than a framework which implements transfer and active learning in two separate stages.

Text categorization, also known as text classification is a supervised classification problem. It aims to assign a predefined class label or group to a new or unknown text document. Most of the time we need a collection of large data from each class to train the classifier. It may be noted that, it is very hard or expensive to collect labelled text data. In most cases we assign the label manually which is neither cost effective nor efficient. In this paper, we have introduced a semi-supervised classification approach where the learner needs very small amount of labelled data with a large amount of unlabeled data to assign a class label to a new or unknown text document. The proposed method uses Kohonen self organizing map (SOM) for labelling the unlabeled data and three classifiers namely support vector machine (SVM), Naive Bayes (NB), and decision tree (DT): classification and regression tree (CART) for observing the accuracy of classification. The experimental results obtained show the effectiveness of our proposed method.

We organized a data mining challenge on “active learning” for IJCNN/WCCI 2010, addressing machine learning problems where labeling data is expensive, but large amounts of unlabeled data are available at low cost. Examples include handwriting and speech recognition, document classification, vision tasks, drug design using recombinant molecules and protein engineering. Such problems might be tackled from different angles: learning from unlabeled data or active learning. In the former case, the algorithms must satisfy themselves with the limited amount of labeled data and capitalize on the unlabeled data with semi-supervised learning methods. Several challenges have addressed this problem in the past. In the latter case, the algorithms may place a limited number of queries to get new sample labels. The goal in that case is to optimize the queries and the problem is referred to as active learning. While the problem of active learning is of great importance, organizing a challenge in that area is non trivial. This is the problem we have addressed, and we describe our approach in this paper. The “active learning” challenge is part of the WCCI 2010 competition program (http://www.wcci2010. org/competition-program). The website of the challenge remains open for submission of new methods beyond the termination of the challenge as a resource for students and researchers (http://clopinet.com/al).

An important challenge for asset management is to analyze large amounts of data in a short period of time to provide insightful information for decision making in a timely fashion. Analyzing all available data manually is impractical and inefficient. It is advantageous to develop pattern recognition algorithms to recognize events-of-interest to achieve effective asset management.Conventional pattern recognition algorithms usually require a fairly large training set in which data points are carefully prepared and "labeled". Examples include designating an equipment’s status as healthy or faulty by subject-matter experts. This can make the process time consuming and error-prone when the training set is large. For most applications, a small amount of data points are already labeled by the experts through their routine activities. While these data points are usually not enough to form a training set for conventional pattern recognition methods, some of the newer methods can take advantage of them along with the hidden manifold structures manifested by the unlabeled data. Moreover, subject matter experts may be willing to provide more input if a clear indication of the value of information and a manageable subset of data is pre-selected for their inspection. In fact, many other industries are facing the same challenge where the cost of acquiring labels is too expensive to be practical and large amounts of unlabeled data and limited expert time are available. A suite of advanced machine learning algorithms (e.g., semi-supervised learning, active learning) have been developed to tackle this challenge, and many of them have been successfully used for various applications in the past few years. In this paper, we will review the concepts and report our observations about the effectiveness of these methods in a real-world asset management scenario. We consider well test validation in an asset with a large number of tests as an example of a label-rich data set that can serve as the basis for our numerical review of existing methods. In this example we will specifically look at the task of building a statistical model to recognize the validity of rate measurement tests in a test separator. In this case, through their daily activities, the operators have labeled most of these tests as valid or invalid. The extensive amount of well test validation data provides sufficient information to assess the newer approaches under review. The plan then is to apply a similar approach to tasks such as equipment health monitoring to identify pump failures with limited expert input.Exxon Mobil Corporation has numerous subsidiaries, many with names that include ExxonMobil, Exxon, Esso and Mobil. For convenience and simplicity in this paper, the parent company and its subsidiaries may be referenced separately or collectively as "ExxonMobil." Abbreviated references describing global or regional operational organizations and global or regional business lines are also sometimes used for convenience and simplicity. Nothing in this paper is intended to override the corporate separateness of these separate legal entities. Working relationships discussed in this paper do not necessarily represent a reporting connection, but may reflect a functional guidance, stewardship, or service relationship.Conceptually, reduction in labeled input can be achieved by combining the information from the labels and the statistical distribution of the data (e.g., clusters). As an extreme example, consider that the pump measurement data may show two distinct clusters and the operators have labeled a few data points in one cluster as pump failures when reports had to be made due to wells being shut in. This information is sufficient to label one of the clusters as healthy and the other one as faulty. For a new measurement, a prediction may be made by first determining the cluster to which the measurement belongs and then assigning it the corresponding label. While most real world problems are much more challenging than this example due to the number of data points, dimensionality of the data, lack of clear cluster structure and potential ambiguity of data structures, similar ideas can be used to develop highly accurate statistical models with a limited number of labels.

Machine learning techniques have shown superior predictive power, among which Bayesian network classifiers (BNCs) have remained of great interest due to its capacity to demonstrate complex dependence relationships. Most traditional BNCs tend to build only one model to fit training instances by analyzing independence between attributes using conditional mutual information. However, for different class labels, the conditional dependence relationships may be different rather than invariant when attributes take different values, which may result in classification bias. To address this issue, we propose a novel framework, called discriminatory target learning, which can be regarded as a tradeoff between probabilistic model learned from unlabeled instance at the uncertain end and that learned from labeled training data at the certain end. The final model can discriminately represent the dependence relationships hidden in unlabeled instance with respect to different possible class labels. Taking k-dependence Bayesian classifier as an example, experimental comparison on 42 publicly available datasets indicated that the final model achieved competitive classification performance compared to state-of-the-art learners such as Random forest and averaged one-dependence estimators.

Representation learning with deep sparse auto-encoder for multi-task learning

The partial area under a receiver operating characteristic curve (pAUC) is a performance measurement for binary classification problems that summarizes the true positive rate with the specific range of the false positive rate. Obtaining classifiers that achieve high pAUC is important in a wide variety of applications, such as cancer screening and spam filtering. Although many methods have been proposed for maximizing the pAUC, existing methods require many labeled data for training. In this paper, we propose a semi-supervised learning method for maximizing the pAUC, which trains a classifier with a small amount of labeled data and a large amount of unlabeled data. To exploit the unlabeled data, we derive two approximations of the pAUC: the first is calculated from positive and unlabeled data, and the second is calculated from negative and unlabeled data. A classifier is trained by maximizing the weighted sum of the two approximations of the pAUC and the pAUC that is calculated from positive and negative data. With experiments using various datasets, we demonstrate that the proposed method achieves higher test pAUCs than existing methods.

We have had to wait over 30 years since the naive Bayes model was first introduced in 1960 for the so-called Bayesian network classifiers to resurge. Based on Bayesian networks, these classifiers have many strengths, like model interpretability, accommodation to complex data and classification problem settings, existence of efficient algorithms for learning and classification tasks, and successful applicability in real-world problems. In this article, we survey the whole set of discrete Bayesian network classifiers devised to date, organized in increasing order of structure complexity: naive Bayes, selective naive Bayes, seminaive Bayes, one-dependence Bayesian classifiers, k -dependence Bayesian classifiers, Bayesian network-augmented naive Bayes, Markov blanket-based Bayesian classifier, unrestricted Bayesian classifiers, and Bayesian multinets. Issues of feature subset selection and generative and discriminative structure and parameter learning are also covered.

The most expressive way humans display emotions is through facial expressions. In this work we report on several advances we have made in building a system for classification of facial expressions from continuous video input. We introduce and test different Bayesian network classifiers for classifying expressions from video. In particular we use Naive-Bayes classifiers and to learn the dependencies among different facial motion features we use Tree-Augmented Naive Bayes (TAN) classifiers. We also investigate a neural network approach. Further, we propose an architecture of hidden Markov models (HMMs) for automatically segmenting and recognizing human facial expression from video sequences. We explore both person-dependent and person-independent recognition of expressions and compare the different methods.

BackgroundIn biomedical articles, a named entity recognition (NER) technique that identifies entity names from texts is an important element for extracting biological knowledge from articles. After NER is applied to articles, the next step is to normalize the identified names into standard concepts (i.e., disease names are mapped to the National Library of Medicine’s Medical Subject Headings disease terms). In biomedical articles, many entity normalization methods rely on domain-specific dictionaries for resolving synonyms and abbreviations. However, the dictionaries are not comprehensive except for some entities such as genes. In recent years, biomedical articles have accumulated rapidly, and neural network-based algorithms that incorporate a large amount of unlabeled data have shown considerable success in several natural language processing problems.ResultsIn this study, we propose an approach for normalizing biological entities, such as disease names and plant names, by using word embeddings to represent semantic spaces. For diseases, training data from the National Center for Biotechnology Information (NCBI) disease corpus and unlabeled data from PubMed abstracts were used to construct word representations. For plants, a training corpus that we manually constructed and unlabeled PubMed abstracts were used to represent word vectors. We showed that the proposed approach performed better than the use of only the training corpus or only the unlabeled data and showed that the normalization accuracy was improved by using our model even when the dictionaries were not comprehensive. We obtained F-scores of 0.808 and 0.690 for normalizing the NCBI disease corpus and manually constructed plant corpus, respectively. We further evaluated our approach using a data set in the disease normalization task of the BioCreative V challenge. When only the disease corpus was used as a dictionary, our approach significantly outperformed the best system of the task.ConclusionsThe proposed approach shows robust performance for normalizing biological entities. The manually constructed plant corpus and the proposed model are available at http://gcancer.org/plantand http://gcancer.org/normalization, respectively.

Infrared ship target segmentation is the important basis of infrared guided weapon in the sea-air context. Typically, accurate infrared ship target segmentation relies on a large number of pixel-level labels. However, it is difficult to obtain them. To this end, we present a method of Semi-supervised Infrared Ship Target Segmentation with Dual Branch (SeISTS-DB), which utilizes a small amount of labeled data and a large amount of unlabeled data to train model and improve segmentation performance. There are three main contributions. First, we design a target segmentation branch to generate the pseudo labels for unlabeled data. It consists of a dual learning network and a segmentation network. The dual learning network generates pseudo labels with weights for unlabeled data. The segmentation network is trained using both labeled data and unlabeled data with pseudo labels to achieve target segmentation of infrared ship, obtaining the preliminary segmentation results. Secondly, we introduce an error segmentation pixel correction branch, which contains a student network and a teacher network, to modify the pixel category error of the preliminary segmentation map. Finally, the outputs of the two branches are combined to obtain the final segmentation result. The SeISTS-DB is compared with other fully-supervised and semi-supervised methods on the infrared ship images dataset. Experimental results demonstrate that when the labeled data accounts for 1/8 of the training data, the mean Intersection over Union (mIou) is respectively improved by 15.35% and 6.19% at most. Besides, it is also compared with other methods on the public IRSTD-1k dataset, when the proportion of labeled images is 1/8, the mIoU is respectively improved by 11.76% at most compared to the state-of-the-art semi-supervised methods, demonstrating its effectiveness.

Intrusion Detection Systems (IDS) are integral to ensuring network security. However, in practical settings, network traffic data often exhibits significant imbalances, affecting both labeled and unlabeled data distributions. Such imbalances notably degrade the performance of existing intrusion detection methods, particularly in semi-supervised learning contexts, where traditional approaches struggle to effectively leverage large amounts of unlabeled data for enhanced detection capabilities. This paper introduces a semi-supervised learning approach based on normalizing flows to mitigate the data imbalance issue in network intrusion detection. Normalizing flows construct flexible and invertible probabilistic models that can accurately capture and generate complex, highdimensional network traffic data distributions. Specifically, this method utilizes a small amount of labeled data for initial training and incorporates manifold learning and self-training with unlabeled data to adapt the model to the imbalance in the unlabeled data distribution, thereby improving overall detection performance. Experimental results demonstrate that this method outperforms traditional approaches in addressing data imbalance in intrusion detection. The proposed method not only improves detection accuracy and recall but also significantly reduces reliance on data distribution assumptions, demonstrating robustness and generalization across diverse network traffic datasets.

Supervised methods for learning sequence classifiers rely on the availability of large amounts of labeled data. However, in many applications because of the high cost and effort involved in labeling the data, the amount of labeled data is quite small compared to the amount of unlabeled data. Hence, there is a growing interest in semi-supervised methods that can exploit large amounts of unlabeled data together with small amounts of labeled data. In this paper, we introduce a novel Abstraction Augmented Markov Model (AAMM) based approach to semi-supervised learning. We investigate the effectiveness of AAMMs in exploiting unlabeled data. We compare semi-supervised AAMMs with: (i) the Markov models (MMs) (which do not take advantage of unlabeled data); and (ii) an expectation maximization (EM) based approach to semi-supervised training of MMs (that make use of unlabeled data). The results of our experiments on three protein subcellular localization prediction tasks show that semi-supervised AAMMs: (i) can effectively exploit unlabeled data; and (ii) are more accurate than both the MMs and the EM based semi-supervised MMs.

We study a class of general M-estimators in the semi-supervised setting, wherein the data are typically a combination of a relatively small labeled dataset and large amounts of unlabeled data. A new estimator, which efficiently uses the useful information contained in the unlabeled data, is proposed via a projection technique. We prove consistency and asymptotic normality, and provide an inference procedure based on K -fold cross-validation. The optimal weights are derived to balance the contributions of the labeled and unlabeled data. It is shown that the proposed method, by taking advantage of the unlabeled data, produces asymptotically more efficient estimation of the target parameters than the supervised counterpart. Supportive numerical evidence is shown in simulation studies. Applications are illustrated in analysis of the homeless data in Los Angeles. Supplementary materials for this article are available online.

Local Feature Extraction (LFE) algorithm is an effective feature extraction method developed in recent years. One of the shortcomings of the current LFE algorithm is that it can only process labeled data, and does not work well when the amount of the labeled data is limited. However, it is usually easy to obtain large amount of unlabeled data but only a few labeled data. In this paper, we propose a new feature extraction algorithm, called Semi-Supervised LFE (SSLFE), which can handle both labeled and unlabeled data to perform feature extraction. In the proposed algorithm, the labeled data are used to maximize the margin and the unlabeled data are used as regulations with respect to the intrinsic geometric structure of the data. The final projection matrix can be obtained by eigenvalue decomposition. Experiments on several datasets demonstrate that SSLFE achieves much higher classification accuracy than LFE.