Semi-Supervised Protein-Protein Interactions Extraction Method Based on Knowledge Distillation and Virtual Adversarial Training

Protein-protein interaction (PPI) plays an extremely vital role in almost all life activities. The study of PPI has always been an important issue in the field of biomedicine. Extracting PPI information from the literature can provide meaningful references for related research. In order to build an automated PPI extraction system, labeled corpora are required. However, labeled corpora are very limited, and annotating corpora is a time-consuming, labor-intensive, and costly task. On the contrary, the amount of unlabeled data is huge and it’s easy to obtain, so it is of great significance to apply semi-supervised learning to PPI extraction. Existing semi-supervised methods have two limitations: 1) cannot make full use of the information in unlabeled data. 2) need to rely on text augmentation methods such as back-translation. Therefore, a semi-supervised PPI extraction method based on label propagation and sentence embedding is proposed in this work. It represents text as numerical features through sentence embedding, and then assigns pseudo-labels to unlabeled data through label propagation, thereby completing semi-supervised training. Experiments on public datasets for PPI extraction show that the proposed method can achieve competitive performance with only a small amount of labeled data. Specifically, when the number of labeled data is 250, it can achieve F1 scores of 36.8% and 53.4% on AIMed and BioInfer, respectively.KeywordsProtein-protein interactionSemi-supervised learningLabel propagationSentence embedding

Medical image annotations are prohibitively time-consuming and expensive to obtain. To alleviate annotation scarcity, many approaches have been developed to efficiently utilize extra information, e.g., semi-supervised learning further exploring plentiful unlabeled data, domain adaptation including multi-modality learning and unsupervised domain adaptation resorting to the prior knowledge from additional modality. In this paper, we aim to investigate the feasibility of simultaneously leveraging abundant unlabeled data and well-established cross-modality data for annotation-efficient medical image segmentation. To this end, we propose a novel semi-supervised domain adaptation approach, namely Dual-Teacher, where the student model not only learns from labeled target data (e.g., CT), but also explores unlabeled target data and labeled source data (e.g., MR) by two teacher models. Specifically, the student model learns the knowledge of unlabeled target data from intra-domain teacher by encouraging prediction consistency, as well as the shape priors embedded in labeled source data from inter-domain teacher via knowledge distillation. Consequently, the student model can effectively exploit the information from all three data resources and comprehensively integrate them to achieve improved performance. We conduct extensive experiments on MM-WHS 2017 dataset and demonstrate that our approach is able to concurrently utilize unlabeled data and cross-modality data with superior performance, outperforming semi-supervised learning and domain adaptation methods with a large margin.

Nowadays deep learning is a tool, which is used in many different pattern recognition problems. For achieving strong results, which are reported in literature, often large amounts of data are necessary. In many use cases collecting or labeling data is very difficult or not possible at all. The aim of this paper is to compare three different algorithmic approaches in order to deal with limited access to labeled and unlabeled data. The drawbacks and benefits of each method are shown and compared to each other. A successful algorithmic approach, which is especially successful for one- and few-shot learning problems, is the usage of external data during the classification task. In this paper siamese neural networks will be investigated in order to evaluate how the usage of external data improves the accuracy. Another widely used approach is consistency regularization. Using consistency regularization state of the art results in semi-supervised learning (SSL) benchmarks are achieved. Virtual adversarial training (VAT) has shown strong results and is chosen as a representative algorithm for consistency regularization. The last approach is the usage of generative adversarial networks (GANs). In literature GANs are often used in order to create additional data and therefor to increase the generalization capability of the classification network. Furthermore the usage of unlabeled data for further performance improvement is considered. The use of unlabeled data is investigated for GANs and VAT.

Automatic detection of retinopathy via computer vision techniques is of great importance for clinical applications. However, traditional deep learning based methods in computer vision require a large amount of labeled data, which are expensive and may not be available in clinical applications. To mitigate this issue, in this paper, we propose a semi-supervised deep learning method built upon pre-trained VGG-16 and virtual adversarial training (VAT) for the detection of retinopathy with optical coherence tomography (OCT) images. It only requires very few labeled and a number of unlabeled OCT images for model training. In experiments, we have evaluated the proposed method on two popular datasets. With only 80 labeled OCT images, the proposed method can achieve classification accuracies of 0.942 and 0.936, sensitivities of 0.942 and 0.936, specificities of 0.971 and 0.979, and AUCs (Area under the ROC Curves) of 0.997 and 0.993 on the two datasets, respectively. When comparing with human experts, it achieves expert level with 80 labeled OCT images and outperforms four out of six experts with 200 labeled OCT images. Furthermore, we also adopt the Gradient Class Activation Map (Grad-CAM) method to visualize the key regions that the proposed method focuses on when making predictions. It shows that the proposed method can accurately recognize the key patterns of the input OCT images when predicting retinopathy.

Many recent text recognition studies achieved successful performance by applying a sequential-label prediction framework such as connectionist temporal classification. Meanwhile, regularization is known to be essential to avoid overfitting when training deep neural networks. Regularization techniques that allow for semi-supervised learning have a greater impact than those that do not. Among widely researched single-label regularization techniques, virtual adversarial training (VAT) performs successfully by smoothing posterior distributions around training data points. However, VAT is almost solely applied to single-label prediction tasks, not to sequential-label prediction tasks. This is because the number of possible candidates in the label sequence exponentially increases with the sequence length, making it impractical to calculate posterior distributions and the divergence between them. Investigating this problem, we have found that there is an easily computable upper bound for divergence. Here, we propose fast distributional smoothing (FDS) as a method for drastically reducing computational costs by minimizing this upper bound. FDS allows regularization at practical computational costs in both supervised and semi-supervised learning. An experiment under simple settings confirmed that upper-bound minimization decreases divergence. Experiments also show that FDS improves scene text recognition performance and enhances state-of-the-art regularization performance. Furthermore, experiments show that FDS enables efficient semi-supervised learning in sequential-label prediction tasks and that it outperforms a conventional semi-supervised method.

The privileged information that is available only for the training examples and not available for test examples, is a new concept proposed by Vapnik and Vashist (Neural Netw 22(5–6):544–557, 2009). With the help of the privileged information, learning using privileged information (LUPI) (Neural Netw 22(5–6):544–557, 2009) can significantly accelerate the speed of learning. However, LUPI is a standard supervised learning method. In fact, in many real-world problems, there are also a lot of unlabeled data. This drives us to solve problems under a semi-supervised learning framework. In this paper, we propose a semi-supervised learning using privileged information (called Semi-LUPI), which can exploit both the distribution information in unlabeled data and privileged information to improve the efficiency of the learning. Furthermore, we also compare the relative importance of both types of information for the learning model. All experiments verify the effectiveness of the proposed method, and simultaneously show that Semi-LUPI can obtain superior performances over traditional supervised and semi-supervised methods.

Multireference (MR) diagnostics are common tools for identifying strongly correlated electronic structure that makes single-reference (SR) methods (e.g., density functional theory or DFT) insufficient for accurate property prediction. However, MR diagnostics typically require computationally demanding correlated wave function theory (WFT) calculations, and diagnostics often disagree or fail to predict MR effects on properties. To overcome these challenges, we introduce a semi-supervised machine learning (ML) approach with virtual adversarial training (VAT) of an MR classifier using 15 WFT and DFT MR diagnostics as inputs. In semi-supervised learning, only the most extreme SR or MR points are labeled, and the remaining point labels are learned. The resulting VAT model outperforms the alternatives, as quantified by the distinct property distributions of SR- and MR-classified molecules. To reduce the cost of generating inputs to the VAT model, we leverage the VAT model's robustness to noisy inputs by replacing WFT MR diagnostics with regression predictions in an MR decision engine workflow that preserves excellent performance. We demonstrate the transferability of our approach to larger molecules and those with distinct chemical composition from the training set. This MR decision engine demonstrates promise as a low-cost, high-accuracy approach to the automatic detection of strong correlation for predictive high-throughput screening.

Automatic extraction of protein-protein interaction (PPI) information from scientific literature is important for building PPI databases, studying biological networks and discovering new biological knowledge through automatic hypothesis generation. In this paper, we present a new method for PPI extraction based on a mixture of logistic models. The method automatically clusters interaction words (words that describe the interactions of protein pairs) into groups with similar grammatical properties. Logistic models are fitted for each cluster of interaction words. Directionality of interactions is an essential piece of information for many protein interactions and important for building directed biological networks. Most of current PPI extraction methods do not extract the directional information of interactions. This is in part due to the lack of specific corpora with directionality information annotated. We introduce a new corpus, PICAD, for evaluating PPI extraction tools that includes directional annotation. The corpus is available at http://stat.fsu.edu/~jinfeng/resources/PICAD.txt. In addition, we propose an ensemble approach using logistic regression, Bayesian Networks, and SVM for identifying PPIs. We show that using an ensemble of classifiers allows us to capture different features in the text and report an F-measure of 75.7% using our new corpus.

Bayesian semi-supervised learning with support vector machine

In machine learning, whether one can build a more accurate classifier by using unlabeled data (semi-supervised learning) is an important issue. Although a number of semi-supervised methods have been proposed, their effectiveness on NLP tasks is not always clear. This paper presents a novel semi-supervised method that employs a learning paradigm which we call structural learning. The idea is to find what good classifiers are like by learning from thousands of automatically generated auxiliary classification problems on unlabeled data. By doing so, the common predictive structure shared by the multiple classification problems can be discovered, which can then be used to improve performance on the target problem. The method produces performance higher than the previous best results on CoNLL'00 syntactic chunking and CoNLL'03 named entity chunking (English and German).

The state-of-the-art methods for protein-protein interaction (PPI) extraction are primarily based on kernel methods, and their performances strongly depend on the handcraft features. In this paper, we tackle PPI extraction by using convolutional neural networks (CNN) and propose a shortest dependency path based CNN (sdpCNN) model. The proposed method (1) only takes the sdp and word embedding as input and (2) could avoid bias from feature selection by using CNN. We performed experiments on standard Aimed and BioInfer datasets, and the experimental results demonstrated that our approach outperformed state-of-the-art kernel based methods. In particular, by tracking the sdpCNN model, we find that sdpCNN could extract key features automatically and it is verified that pretrained word embedding is crucial in PPI task.

Mining Protein-Protein Interactions (PPIs) from the fast-growing biomedical literature resources has been proven as an effective approach for the identification of biological regulatory networks. This paper presents a novel method based on the idea of Interaction Relation Ontology (IRO), which specifies and organises words of various proteins interaction relationships. Our method is a two-stage PPI extraction method. At first, IRO is applied in a binary classifier to determine whether sentences contain a relation or not. Then, IRO is taken to guide PPI extraction by building sentence dependency parse tree. Comprehensive and quantitative evaluations and detailed analyses are used to demonstrate the significant performance of IRO on relation sentences classification and PPI extraction. Our PPI extraction method yielded a recall of around 80% and 90% and an F1 of around 54% and 66% on corpora of AIMed and BioInfer, respectively, which are superior to most existing extraction methods.

Semi-supervised learning acts as an effective way to leverage massive unlabeled data. In this paper, we propose a novel training strategy, termed as Semi-supervised Contrastive Learning (SsCL), which combines the well-known contrastive loss in self-supervised learning with the cross entropy loss in semi-supervised learning, and jointly optimizes the two objectives in an end-to-end way. The highlight is that different from self-training based semi-supervised learning that conducts prediction and retraining over the same model weights, SsCL interchanges the predictions over the unlabeled data between the two branches, and thus formulates a co-calibration procedure, which we find is beneficial for better prediction and avoid being trapped in local minimum. Towards this goal, the contrastive loss branch models pairwise similarities among samples, using the pseudo labels generated from the cross entropy branch, and in turn calibrates the prediction distribution of the cross entropy branch with the contrastive similarity. We show that SsCL produces more discriminative representation and is beneficial to semi-supervised learning. Notably, on ImageNet with ResNet50 as the backbone, SsCL achieves 60.2% and 72.1% top-1 accuracy with 1% and 10% labeled samples, respectively, which significantly outperforms the baseline, and is better than previous semi-supervised and self-supervised methods.

Semi-supervised trees for multi-target regression

Feature representation is essential to machine learning and text mining. In this paper, we present a feature coupling generalization (FCG) framework for generating new features from unlabeled data. It selects two special types of features, i.e., example-distinguishing features (EDFs) and class-distinguishing features (CDFs) from original feature set, and then generalizes EDFs into higher-level features based on their coupling degrees with CDFs in unlabeled data. The advantage is: EDFs with extreme sparsity in labeled data can be enriched by their co-occurrences with CDFs in unlabeled data so that the performance of these low-frequency features can be greatly boosted and new information from unlabeled can be incorporated. We apply this approach to three tasks in biomedical literature mining: gene named entity recognition (NER), protein-protein interaction extraction (PPIE), and text classification (TC) for gene ontology (GO) annotation. New features are generated from over 20 GB unlabeled PubMed abstracts. The experimental results on BioCreative 2, AIMED corpus, and TREC 2005 Genomics Track show that 1) FCG can utilize well the sparse features ignored by supervised learning. 2) It improves the performance of supervised baselines by 7.8 percent, 5.0 percent, and 5.8 percent, respectively, in the tree tasks. 3) Our methods achieve 89.1, 64.5 F-score, and 60.1 normalized utility on the three benchmark data sets.