ASSET: A Semi-supervised Approach for Entity Typing in Knowledge Graphs

Most of the online queries target entities and the type of an entity is a key piece of information. Entity type helps us to understand what an entity is and how it relates to other entities mentioned in a document. Search engine result pages (SERPs) often surface facts and entity type information from a background Knowledge Graph (KG) in response to queries that carry a semantic information need. In a KG, an entity usually holds multiple type properties. For example, popular types attached to the entity `Donald Trump' via rdfs:type statements might be Person, Businessman, and Leader. However, other types of this entity, e.g., Solicitor, Restaurateur, and Writer might also be interesting to some users. Unpopular entity types can be useful for tail queries like, for example, `Is Donald Trump an American Television Producer' or `Is Donald Trump an American Television Actor'. It is then important to, given an entity in a KG, rank entity types attached to the entity by relevance to a certain user and information need as not always the most popular type is the most informative within a textual context. In this paper we address the entity type ranking problem by means of KG embedding models. In our work, we show that entity type ranking can be seen as a special case of the KG completion problem. Embeddings can be learned from the structural, probabilistic and contextual description information of the entities. We propose and evaluate our methods to find the most relevant entity type based on collection statistics and on the graph structure interconnecting entities and types. Experimental results show that our proposed approaches outperform the state-of-the-art type ranking models while, at the same time, being more efficient and scalable. Our approach focuses on the task of ranking a set of types associated to an entity in a background knowledge graph to select the most relevant types.

Knowledge enhancement-based large-scale natural language training model is an advanced language model that combines deep learning and knowledge enhancement. By learning from massive unlabeled data and combining with external knowledge such as knowledge graph, it breaks through the limitations of traditional models in interpretability and reasoning ability. Introducing knowledge into data-driven artificial intelligence model is an important way to realize human-machine hybrid intelligence. However, since most pre-trained models are trained on large-scale unstructured corpus data, the defects in certainty and explainability can be remedied to some extent by introducing external knowledge. To solve the above problems, we present a knowledge-enhanced large-scale natural language training model that integrates deep learning with external knowledge sources (e.g., knowledge graphs) to improve interpretability and reasoning ability. This approach addresses the limitations of traditional models trained on unstructured data by incorporating external knowledge to enhance certainty and explainability. We propose a new knowledge enhancement method and demonstrate its effectiveness through a long text representation model. This model processes structured, knowledge-rich long texts by extracting and integrating knowledge and semantic information at the sentence and document levels. It then fuses these representations to generate an enhanced long text representation. Experiments on legal case matching tasks show that our model significantly outperforms existing methods, highlighting its innovation and practical value.

Active Learning is a machine learning and data mining technique that selects the most informative samples for labeling and uses them as training data; it is especially useful when there are large amount of unlabeled data and labeling them is expensive. Recently, batch-mode active learning, where a set of samples are selected concurrently for labeling, based on their collective merit, has attracted a lot of attention. The objective of batch-mode active learning is to select a set of informative samples so that a classifier learned on these samples has good generalization performance on the unlabeled data. Most of the existing batch-mode active learning methodologies try to achieve this by selecting samples based on certain criteria. In this article we propose a novel criterion which achieves good generalization performance of a classifier by specifically selecting a set of query samples that minimize the difference in distribution between the labeled and the unlabeled data, after annotation. We explicitly measure this difference based on all candidate subsets of the unlabeled data and select the best subset. The proposed objective is an NP-hard integer programming optimization problem. We provide two optimization techniques to solve this problem. In the first one, the problem is transformed into a convex quadratic programming problem and in the second method the problem is transformed into a linear programming problem. Our empirical studies using publicly available UCI datasets and two biomedical image databases demonstrate the effectiveness of the proposed approach in comparison with the state-of-the-art batch-mode active learning methods. We also present two extensions of the proposed approach, which incorporate uncertainty of the predicted labels of the unlabeled data and transfer learning in the proposed formulation. In addition, we present a joint optimization framework for performing both transfer and active learning simultaneously unlike the existing approaches of learning in two separate stages, that is, typically, transfer learning followed by active learning. We specifically minimize a common objective of reducing distribution difference between the domain adapted source, the queried and labeled samples and the rest of the unlabeled target domain data. Our empirical studies on two biomedical image databases and on a publicly available 20 Newsgroups dataset show that incorporation of uncertainty information and transfer learning further improves the performance of the proposed active learning based classifier. Our empirical studies also show that the proposed transfer-active method based on the joint optimization framework performs significantly better than a framework which implements transfer and active learning in two separate stages.

Representation learning with deep sparse auto-encoder for multi-task learning

Knowledge graph (KG) embedding aims to study the embedding representation to retain the inherent structure of KGs. Graph neural networks (GNNs), as an effective graph representation technique, have shown impressive performance in learning graph embedding. However, KGs have an intrinsic property of heterogeneity, which contains various types of entities and relations. How to address complex graph data and aggregate multiple types of semantic information simultaneously is a critical issue. In this article, a novel heterogeneous GNNs framework based on attention mechanism is proposed. Specifically, the neighbor features of an entity are first aggregated under each relation-path. Then the importance of different relation-paths is learned through the relation features. Finally, each relation-path-based features with the learned weight values are aggregated to generate the embedding representation. Thus, the proposed method not only aggregates entity features from different semantic aspects but also allocates appropriate weights to them. This method can capture various types of semantic information and selectively aggregate informative features. The experiment results on three real-world KGs demonstrate superior performance when compared with several state-of-the-art methods.

Intrusion Detection Systems (IDS) are integral to ensuring network security. However, in practical settings, network traffic data often exhibits significant imbalances, affecting both labeled and unlabeled data distributions. Such imbalances notably degrade the performance of existing intrusion detection methods, particularly in semi-supervised learning contexts, where traditional approaches struggle to effectively leverage large amounts of unlabeled data for enhanced detection capabilities. This paper introduces a semi-supervised learning approach based on normalizing flows to mitigate the data imbalance issue in network intrusion detection. Normalizing flows construct flexible and invertible probabilistic models that can accurately capture and generate complex, highdimensional network traffic data distributions. Specifically, this method utilizes a small amount of labeled data for initial training and incorporates manifold learning and self-training with unlabeled data to adapt the model to the imbalance in the unlabeled data distribution, thereby improving overall detection performance. Experimental results demonstrate that this method outperforms traditional approaches in addressing data imbalance in intrusion detection. The proposed method not only improves detection accuracy and recall but also significantly reduces reliance on data distribution assumptions, demonstrating robustness and generalization across diverse network traffic datasets.

The partial area under a receiver operating characteristic curve (pAUC) is a performance measurement for binary classification problems that summarizes the true positive rate with the specific range of the false positive rate. Obtaining classifiers that achieve high pAUC is important in a wide variety of applications, such as cancer screening and spam filtering. Although many methods have been proposed for maximizing the pAUC, existing methods require many labeled data for training. In this paper, we propose a semi-supervised learning method for maximizing the pAUC, which trains a classifier with a small amount of labeled data and a large amount of unlabeled data. To exploit the unlabeled data, we derive two approximations of the pAUC: the first is calculated from positive and unlabeled data, and the second is calculated from negative and unlabeled data. A classifier is trained by maximizing the weighted sum of the two approximations of the pAUC and the pAUC that is calculated from positive and negative data. With experiments using various datasets, we demonstrate that the proposed method achieves higher test pAUCs than existing methods.

This systematic literature review investigates how evolvable knowledge graphs (KGs) enhance manufacturing. Evolvable KGs adapt to changing knowledge by leveraging machine learning algorithms and human expertise to improve decision-making, operational efficiency, and predictive maintenance capabilities beyond the capabilities of prevailing solutions enabled by dynamic KGs. This review maps the existing literature based on the stages of the KG development process. The main results are an updated model of the creation and maintenance of evolvable KGs in manufacturing and a literature-based synthesis of learning approaches for enabling KG evolution through knowledge processing and knowledge utilization. The results contribute to updating the KG development process model and synthesizing an understanding of evolvable KGs that utilize the observed learning approaches on knowledge processing to change KG relations and nodes, and three learning approaches through knowledge utilization (human-guided, machine-driven, and human-machine collaborative knowledge updates). Despite emerging advances, challenges persist in quality assurance, process planning, and integration of human expertise. The findings advocate addressing these issues to promote greater adoption and optimization of KGs in manufacturing. By deepening the understanding of how KGs can evolve through learning, this review sets a conceptual basis for future research to develop more dynamic and intelligent KG-based manufacturing systems.

Training classification models in the medical domain is often difficult due to data heterogeneity (related to acquisition procedures) and due to the difficulty of getting sufficient amounts of annotations from specialized experts. It is particularly true in digital pathology, where models do not generalize easily. This paper presents a novel approach for the generalization of models in conditions where heterogeneity is high and annotations are few. The approach relies on a semi-supervised teacher/student paradigm to different datasets and annotations. The paradigm combines a small amount of strongly-annotated data, with a large amount of unlabeled data, for training two Convolutional Neural Networks (CNN): the teacher and the student model. The teacher model is trained with strong labels and used to generate pseudo-labeled samples from the unlabeled data. The student model is trained combining the pseudo-labeled samples and a small amount of strongly-annotated data. The paradigm is evaluated on the student model performance of Gleason pattern and Gleason score classification in prostate cancer images and compared with a fully-supervised learning approach for training the student model. In order to evaluate the capability of the approach to generalize, the datasets used are highly heterogeneous in visual characteristics and are collected from two different medical institutions. The models, trained with the teacher/student paradigm, show an improvement in performance above the fully-supervised training. The models generalize better on both the datasets, despite the inter-datasets heterogeneity, alleviating the overfitting. The classification performance shows an improvement both in the classification of Gleason pattern at patch level (\(\kappa \,=\,0.6129\, \pm \, 0.0127\) from \(\kappa \,=\,0.5608\pm 0.0308\)) and at in Gleason score classification, evaluated at WSI-level \(\kappa \,=\,0.4477\pm 0.0460\) from \(\kappa \,=\,0.2814\pm 0.1312\)).

Abstract. Natural disasters have a significant impact on the environment and economies of all countries around the world, and a large amount of multi-source heterogeneous geographic information data is generated every day. However, due to a lack of knowledge transformation capabilities, these nations continue to struggle with the issue of "a large amount of data and little knowledge". Therefore, it is of great significance how to extract geographic knowledge related to disasters from the vast data and construct a geographic knowledge graph integrating disaster information. Based on the theory related to knowledge extraction, this paper proposes a method to construct a natural disaster knowledge graph integrating geographic information. The core of this knowledge graph is to construct the association relationship between natural disaster concepts, research areas, and spatial data. The vocabulary and relationships associated with disaster concepts are primarily transformed by an existing word list of geographic narratives, which then provide rich semantic relationships of domain concepts for the entire knowledge graph. The research areas and spatial data types are mainly obtained through knowledge entity extraction and disambiguation methods. This disaster knowledge graph can support applications well such as natural disaster visualization and analysis, data recommendation systems, and intelligent Q&A systems, which can further improve the intelligence of natural disaster knowledge services and is expected to promote the sharing and reuse of domain knowledge graphs to a certain extent.

Developing ladder network for intelligent evaluation system: Case of remaining useful life prediction for centrifugal pumps

Understanding human emotions is one of the necessary skills for the computer to interact intelligently with human users. The most expressive way humans display emotions is through facial expressions. In this paper, we report on several advances we have made in building a system for classification of facial expressions from continuous video input. We use Bayesian network classifiers for classifying expressions from video. One of the motivating factor in using the Bayesian network classifiers is their ability to handle missing data, both during inference and training. In particular, we are interested in the problem of learning with both labeled and unlabeled data. We show that when using unlabeled data to learn classifiers, using correct modeling assumptions is critical for achieving improved classification performance. Motivated by this, we introduce a classification driven stochastic structure search algorithm for learning the structure of Bayesian network classifiers. We show that with moderate size labeled training sets and large amount of unlabeled data, our method can utilize unlabeled data to improve classification performance. We also provide results using the Naive Bayes (NB) and the Tree-Augmented Naive Bayes (TAN) classifiers, showing that the two can achieve good performance with labeled training sets, but perform poorly when unlabeled data are added to the training set.

Knowledge Graphs (KG) represent a large amount of Semantic Associations (SAs), i.e., chains of relations that may reveal interesting and unknown connections between different types of entities. Applications for the contextual exploration of KGs help users explore information extracted from a KG, including SAs, while they are reading an input text. Because of the large number of SAs that can be extracted from a text, a first challenge in these applications is to effectively determine which SAs are most interesting to the users, defining a suitable ranking function over SAs. However, since different users may have different interests, an additional challenge is to personalize this ranking function to match individual users’ preferences. In this paper we introduce a novel active learning to rank model to let a user rate small samples of SAs, which are used to iteratively learn a personalized ranking function. Experiments conducted with two data sets show that the approach is able to improve the quality of the ranking function with a limited number of user interactions.

An important challenge for asset management is to analyze large amounts of data in a short period of time to provide insightful information for decision making in a timely fashion. Analyzing all available data manually is impractical and inefficient. It is advantageous to develop pattern recognition algorithms to recognize events-of-interest to achieve effective asset management.Conventional pattern recognition algorithms usually require a fairly large training set in which data points are carefully prepared and "labeled". Examples include designating an equipment’s status as healthy or faulty by subject-matter experts. This can make the process time consuming and error-prone when the training set is large. For most applications, a small amount of data points are already labeled by the experts through their routine activities. While these data points are usually not enough to form a training set for conventional pattern recognition methods, some of the newer methods can take advantage of them along with the hidden manifold structures manifested by the unlabeled data. Moreover, subject matter experts may be willing to provide more input if a clear indication of the value of information and a manageable subset of data is pre-selected for their inspection. In fact, many other industries are facing the same challenge where the cost of acquiring labels is too expensive to be practical and large amounts of unlabeled data and limited expert time are available. A suite of advanced machine learning algorithms (e.g., semi-supervised learning, active learning) have been developed to tackle this challenge, and many of them have been successfully used for various applications in the past few years. In this paper, we will review the concepts and report our observations about the effectiveness of these methods in a real-world asset management scenario. We consider well test validation in an asset with a large number of tests as an example of a label-rich data set that can serve as the basis for our numerical review of existing methods. In this example we will specifically look at the task of building a statistical model to recognize the validity of rate measurement tests in a test separator. In this case, through their daily activities, the operators have labeled most of these tests as valid or invalid. The extensive amount of well test validation data provides sufficient information to assess the newer approaches under review. The plan then is to apply a similar approach to tasks such as equipment health monitoring to identify pump failures with limited expert input.Exxon Mobil Corporation has numerous subsidiaries, many with names that include ExxonMobil, Exxon, Esso and Mobil. For convenience and simplicity in this paper, the parent company and its subsidiaries may be referenced separately or collectively as "ExxonMobil." Abbreviated references describing global or regional operational organizations and global or regional business lines are also sometimes used for convenience and simplicity. Nothing in this paper is intended to override the corporate separateness of these separate legal entities. Working relationships discussed in this paper do not necessarily represent a reporting connection, but may reflect a functional guidance, stewardship, or service relationship.Conceptually, reduction in labeled input can be achieved by combining the information from the labels and the statistical distribution of the data (e.g., clusters). As an extreme example, consider that the pump measurement data may show two distinct clusters and the operators have labeled a few data points in one cluster as pump failures when reports had to be made due to wells being shut in. This information is sufficient to label one of the clusters as healthy and the other one as faulty. For a new measurement, a prediction may be made by first determining the cluster to which the measurement belongs and then assigning it the corresponding label. While most real world problems are much more challenging than this example due to the number of data points, dimensionality of the data, lack of clear cluster structure and potential ambiguity of data structures, similar ideas can be used to develop highly accurate statistical models with a limited number of labels.

Semi-Supervised Learning (SSL) with mismatched classes deals with the problem that the classes-of-interests in the limited labeled data are only a subset of the classes in massive unlabeled data. As a result, classical SSL methods would be misled by the classes which are only possessed by the unlabeled data. To solve this problem, some recent methods divide unlabeled data to useful in-distribution (ID) data and harmful out-of-distribution (OOD) data, among which the latter should particularly be weakened. As a result, the potential value contained by OOD data is largely overlooked. To remedy this defect, this paper proposes a “Transferable OOD data Recycling” (TOOR) method which properly utilizes ID data as well as the “recyclable” OOD data to enrich the information for conducting class-mismatched SSL. Specifically, TOOR treats the OOD data that have a close relationship with ID data and labeled data as recyclable, and employs adversarial domain adaptation to project them to the space of ID data and labeled data. In other words, the recyclability of an OOD datum is evaluated by its transferability, and the recyclable OOD data are transferred so that they are compatible with the distribution of known classes-of-interests. Consequently, our TOOR extracts more information from unlabeled data than existing methods, so it achieves an improved performance which is demonstrated by the experiments on typical benchmark datasets.